foreign
I'm gonna start my introduction with a
secret about myself
I actually really don't fit in here
I have no professional training in
information technology I have no degree
in computer science
I don't have anything professional in my
background to deal with computers in any
way at all
but I'm gonna do my best with you guys
today
and share with you some of my personal
journey and a fantastic Journey on data
engineering
I'm actually a geologist
and I've been on this journey to find
out how this
cool thing that apparently you guys do
call data engineering mixes with what I
know which is the geosciences
my name is Ashley Russell
I'm actually American but I have lived
in Norway for several years now I live
in savanga on the other side of Norway
I'm here speaking on behalf of Ecuador
obviously the logo is on the slide for
those who are Norwegian you probably
know who we are those of you who are not
Norwegian we are the National Energy
company of Norway
and yes our primary business is oil and
gas
and if any of you are aware of the
current volatile situation in Europe we
have a major duty to supply for example
gas to the European continent right now
but times are changing
and we are also heavily investing in
renewable energy and things like
offshore wind
um and carbon capture and I'll come back
to a little bit of that today
no
if you would ask me in 2015
when I joined Equinox some 20 something
year old very bright-eyed bushy-tailed
if I knew how a computer worked
I would have said no and it's not like
okay I wasn't born in the woods or
something right I could work a computer
but I took it for granted right I turned
it on clicked on something opened Excel
and everything was fine until you tried
loading 5000 rows of data into Excel
right
today I probably know more about
computers than I really should integrate
with a degree in geology
um because I'm also a self-taught data
scientist
and one of the first light bulb moments
for me was when I was first learning
data science and learning to code in
Python was my gateway drug to this whole
thing and reason why I'm here today but
I began to automate with python a lot of
sort of boring file maintenance and data
management stuff right just making my
job a little bit easier
and the light bulb kind of dawned on me
and like oh my gosh like this could
really revolutionize a lot of the awful
data crap that we deal with
and I didn't know the word for it at the
time right but now today we have a very
nice term for this kind of concept
called Data engineering
now Equinox I know we have a nice Booth
downstairs I suppose I can ask for you
who know about our company what you
might think about our digital capability
or skill sets in information technology
and maybe you think about something like
this
this is the perhaps the most heavily
used software in all of Ecuador it's
called sap
now I remember okay I remember joining
Equinox right and having to go write my
hours my hours for the month for the
first time and opening this application
and staring dumb struck at the screen
like I've literally been transported
backward in time to the year 1993. where
am I right
so we tend to have this perception in
the computer science world of being
fairly traditional quite risk adverse
when it comes to quote unquote
information technology
today I hope to change your perception a
little bit
and to introduce how we are finding our
way as a broad Energy company not just
an oil and gas but carbon capture in
Wind too
and by automating data flow data
Corrections and accessibility data so
our data science applications
Partnerships with technology companies
like the ones that you are all
representing and that can flourish and
we transform to a long lasting highly
digital business that it stealth has
data that can move also Beyond oil and
gas
now it's not all doom and gloom okay sap
does not mean everything we do have some
key focuses in sort of the quote-unquote
digital Realm
um first is that we have a really strong
commitment in Ecuador to open source we
release both data openly for use and we
also release the software I'll share
some links at the end of the
presentation today
the second is what we would say the
diamond in the crown of Equinox this is
an oil gas field called the Johannes
fedra now that's not the important part
the important part is that Johan's
Fedrick was planned for and executed as
a what we call digital front runner it's
got some wicked cool Tech
on the top side in the facility we have
everything from iot to predictive
maintenance to AR to full digital twins
all happening
now also below the sea bottom in the
subsurface with Johan startup we also
have a full subsurface team with
integrated software development directly
in the business
so it's a great place for us to
experiment work with a lot of data
there's a lot of sensor data here and
find new ways of working with quantities
of data that we can maybe take to other
places too
the last one is the one that I'm
obviously really talking about today is
that we have a strong relationship with
Microsoft we are a Azure shop
and it goes just beyond providing Azure
as a you know a cloud resource for us
but also partnering on projects trying
new things giving up support and this
extends across not of course our oil and
gas business but also into New
Opportunities including offshore wind
and carbon capture as well
and so for example we uh just last week
worked together and Microsoft released
the new energy data services so yeah
yeah we're going to focus mostly on
that last
today
okay
we're going to be talking about a
particular business area that I work in
called sub surface
to talk about the subsurface we need to
go back to geology 101.
I'm sorry if any of you have not had a
geology 101 class I promise this is
going to be simple
and then Pro and it really is there's
only three types of rocks
okay that's it we're quite easy right
um and let's see how much you guys know
about this rock
so raise the hands three types of rocks
all those who think it's metamorphic
ah
anyone who thinks this is a volcanic
rock
oh man you guys are maybe more experts
than I intended
all right how many you think it's a
sedimentary rock yeah okay wow you guys
really know your geology
so essentially the subsurface is made up
of three these three different kinds of
rocks that's it I mean of course there's
a lot of subcategories and divisions
here don't get me wrong it's quite
complex but basically that's it
now
we see a lot of rocks of course exposed
to us on the surface you go for a walk
in the park and you see a nice Cliff for
outcropping a rock right all the way to
the high Himalaya
but for those rocks that are for example
below our feet
or below the ocean floor
we can't go directly see or feel or
touch those rocks
we cannot actually say with exact
certainty what they are
now that makes for a very interesting
situation
yeah it's an enter
okay let me back up a little bit to the
Ecuador side now we have a three-prong
strategy to become a broad Energy
company
that includes Renewables we're very
strong enough for our wind it includes
of course oil and gas and low carbon for
meat and subsurface this really points
to what we call carbon capture and
sequestration we take carbon dioxide out
of the atmosphere and we put it back
into the Rocks into the subsurface to
store it long term
now
understanding the rocks or the
composition of the subsurface is
important for all three of those things
okay even take a big wind turbine right
you're putting a big offshore wind
turbine in the middle of the ocean you
better know what the makeup of the Rocks
supporting the foundational support of
that wind turbine are
so this means we need to have a very
good understanding of the subsurface
now how do we do that right well of
course we take samples
so we have a couple ways that we can
take samples of the rocks in the
subsurface
the first is we can drill a well and
when we drill a well we can acquire some
data within it
very rarely do we take direct samples we
can it's very expensive and has a high
risk of failure
so we take a lot of data Within These
Wells by Direct by using stimuli or by
proxy
additionally we can do something called
seismic
this is essentially sound Imaging where
we shoot sound waves into the layers of
rock in the subsurface and record the
refractions coming back
it tells us something about the
different compositions of the Rocks the
different types of them they refract
sound in different ways
now we're going to look at the opposite
of this situation for a minute
we're in a very different type of data
acquisition world when we talk about
subsurface but imagine
you're here this is actually Nord marker
it's just a short T Bonner ride north of
here but not slow it's a beautiful
Forest many lakes beautiful trees you're
here you can smell the pine in your nose
the wind and your hair it's a much more
beautiful warm day than today
but if you think about this environment
there's a lot of ways that you can
directly measure what it is you bring
along your handy thermometer you can
directly measure the air temperature in
the water temperature
you could walk around the lake and
actually measure its circumference right
you could even catch a fish and record
the species of that fish
we are in a very different situation
when we talk about trying to understand
the composition of the rocks in the
subsurface
now even more I don't know terrifying is
that it underpins all three of our main
businesses in Ecuador is understanding
what the makeup of those rocks are
It Is by its own nature a predictive
business
and we're a multi-billion dollar Revenue
company
so it's pretty wild right we better get
it right no we don't always get it right
now
through to today the interpretation of
this collected data has been done
through a lot of manual work a lot of
expertise in trying to do their best to
integrate data to make a best guess of
what the rocks are going to look like
given a specific location
um so if you're like me if any of you
are data scientists in the room you're
probably thinking okay this is a
predictive business can we get better at
making those predictions
can we do it faster can we do it better
can we have more handling of the
possible outcomes and uncertainties
and you would be right you know concepts
of machine learning are very applicable
here and we'll talk a bit about that
but
many of you who have worked with any
type of analytics or data science will
know that that only works as well as the
data that they are based upon
and someone introduces our challenge
today
because the subsurface is something we
cannot easily or directly measure we
take many different types of samples
most of them are taken through various
stimuli
now this results in a whole swarm of
different kinds of data data products
data sets data types whatever you want
whatever buzzword you want to use for
this right
But Here Comes the important and fun
part
each one of these different types of
data comes with their own unique file
formats physical file size concepts of
metadata concepts of quality and
completely different formats and
structures
and just to point out there's not a ton
of Time series data here yeah we have
some but we're in a very different more
static world when we talk about
subsurface data so there's not a ton of
like out of the box like time series
analysis sort of tools that we can
really use here
so we decided an equinox subsurface with
a start with improvements to just one of
these
how can we understand how to do proper
modern data engineering by picking off
perhaps a low hanging fruit we'll see if
that's actually true
and so we decided to work with a type of
data called wall logs it's got a few
things going for it one it's tabular
data awesome rows and columns we love it
we collect a ton of it so for data
science it's a really great
statistically large data set awesome
and third it's one of the most heavily
used types of data in the subsurface
organization if you talk to anyone no
matter their scientific background or
discipline they know and use while logs
in their daily work
so we decided we start here
now well logging in itself I just have
to give you a little bit of information
about what it actually is it's actually
really amazing technology in its own
right
now some French guys in the Paris Basin
outside of Paris France you may
recognize the name on that truck there I
think they're also in the Expo
downstairs
but these French guys in Terrace Basin
in Mark of the year 1927.
decided to try making an electric
current from inside a drilled well
through the Rocks outside that well and
record the changes in conductivity or
the inverse resistivity to try to
understand if they could see differences
in the Rocks they're looking for
remember our friend sandstone
now long story short it works very well
and with the technology advances from
World War II mostly on the atomic front
right well logging today isn't just
about electric signals but we also use a
whole variety of different stimuli to
try to understand what the rocks are
like along a wild boar
so we use things like light we use
things like neutrons gamma rays and we
also use some mechanical measurements
but the point being this started in
1927. that's a hundred years of data
acquisition of whitlocks
so needless to say as you can imagine
there's a lot of legacy
contained in well-logged data that we as
humans have had been dealing with for
the past several well several decades to
make a proper analysis
now traditionally the way that this data
is handled is that when we complete a
well we've hired some type of vendor
and they've done the well logging right
so they have a much nicer boat or truck
now right and they deliver to us a
series of files we can have a couple
or they might give us a whole bunch it
depends on what kind of acquisition
program we've decided to have
now
a person essentially loads all these
files into a special oil and gas
software this is a traditional method
and begins to work and make a lot of
Transformations within that GUI now
these might be things like removing
outliers you know Common Transformations
removing sections of bad quality maybe
they do some unit of measure
transformation some schema changes to
the naming you know pretty conventional
things
they'll also add a little bit of the
special Ecuador sauce into it too to do
some calculations as well to give the
data a little bit more of uh something
that gives people a little more of an
indication of how to use it
now
typically this is only done for Wells
that are deemed important in some way
okay we drill well we don't find
anything you've sunk a lot of cost into
that we're not going to keep working on
it right
and of course that creates a very nice
for us data scientist a inherently
biased data set
not only that to make matters worse
the Transformations that these folks are
doing in this proprietary software are
not recorded anywhere
so what happens is this person does all
these Transformations and they produce
the master the composite the one file
per well that everyone
should use
but we have no lineage information on
how that was actually created
cool right you can imagine the grief and
rework that this has caused us
um to make matters even a little bit
more worse
that's a person doing a lot of
Transformations here everyone has a
little different Flair to how they do
that and it's changed over time
so the whole process here takes about a
week or two and if we Outsource it which
we do sometimes you're talking between a
thousand and three thousand US dollars
to do this for one well that's actually
quite expensive
um obviously here in live the
opportunity
if we can do these sort of
Transformations more automatically and
we don't care about which data we can do
it for any day we like
we record lineage so we know how things
were done
can we have more standardized data
faster right with the ability to
understand exactly what had created that
Master file
so
that was the opportunity that we were
talking about here
now I know you guys are pretty technical
what does Well Log data actually look
like and
in reality this master composite looks
like this on the left
it's essentially like I said a row in
column structure with a metadata section
at the top the fun part is that this is
in a standard called dot Las a last file
which is a industry file it's only in
the oil and gas industry you won't find
it anywhere else right so that's not
very great for things like azure
um but essentially
hopefully in the metadata section we
have some aspect about what each of the
columns are if we have something about
the units of measure and typically some
type of short descriptions we have a
little bit of metadata to help us out
now on the right
is what well logs look like when we
actually style them so we add some line
coloring some some fills and this is
what a geoscientist would typically work
with
so if you showed this slide to say a
general geoscientist or geologist in
Ecuador and ask them to pick out the
well logs they would instantly know oh
yeah that picture on the right those are
those are Well Log data data but it
would take them quite a bit of time to
understand that the picture on the left
is the same data
the point being anything we do to
automate the preparation of that file
the end result needs to behave
like the image on the right
okay
don't worry too much about the slide
just map Ecuador drills and has drilled
a lot of Wells worldwide we've had a lot
of opportunities for data collection put
it that way this is a pure marketing
slide Capital markets update numbers
don't matter
but the point being is that we have a
variety of projects across the world we
are not just in Norway
we of course have a lot of focus in
Norway but that's not it
so we have data from all over pretty
much
multiple countries across the world we
also have access to publicly released
data which is nice most countries
release their well data after a certain
number of years we can also trade with
other companies or we purchase data from
an external vendor
so if we take the business opportunity I
just mentioned to try to automate and
standardize the sort of transformation
that people are doing physically today
and taking all those many many different
files per well and turning them into V
Master the composite
if you could do this globally
right we have about 120
000 Wells in-house that we have data for
each one has about 50 well logs 50
columns in each file that means we're
looking at a pretty massive count of
data which is great for data science
right about 6 million well logs or 6
million of those columns
now
for any of you who work with things like
mlops or any type of machine learning
model and trying to scale them from one
project to another right you can imagine
the power that this has for data science
globally you have perfectly standardized
files and every geographical location
for every project you are working in
right
data science across our subsurface
projects worldwide we can make
predictions faster more comprehensive
right and with more probabilistic
results right we can handle things like
Risk in a little bit better way
and the key point being is that we
prevent the data scientists and each of
those project locations from each having
to do their own data wrangling of well
logs right which means we can move
a lot faster
you can probably guess what the next
word out of my mouth is going to be
but
there are some very good reasons that
we've been doing this processing of this
data manually for so many years
life is never that simple and I think
you guys know that in this
so there are four key challenges that we
had to figure out how to engineer and if
the data was really bad at least find
some way to have good logging so that we
could handle it manually
okay I don't know any Brits in the
audience
anyone know where this is
this is London the Millennium Bridge
and the first challenge revolves around
units of measure I'm sure something many
of you have probably struggled with
yourselves
and a little quiz for you you can go
ahead and Google it if you like did the
United Kingdom complete its conversion
to metric units
the answer is no there's a very long and
winded article on Wikipedia about this
if you have you know 20 minutes to read
it but I think the best way to explain
it is that there exists in the UK today
an advocacy group called the active
resistance to metrication I'm not
kidding it literally this exists
a brexit thing out of yeah
so what this means for us doing data
engineering of Well Log data is that the
UK is extremely unpredictable in which
unit measure systems that we're dealing
with
and to make things even worse we find
situations where the template for the
acquisition program was set up in one
unit measure system in Imperial for
example but the actual data collection
was done in metric
really fun
now this problem goes worldwide
the number of unit systems we deal with
in while logging is actually
mind-boggling we use so many different
types of measure systems we're using a
pretty wide variety of different stimuli
to collect data
um and not only that but as you guys are
also probably aware the way that you can
spell units of measure can be very
different from one place to another
so something you think as simple as
inches we found for example six
different ways that you can write the
word inches and if the unit measure
includes a symbol of some kind God
forbid a fraction or an exponent then we
really have a big problem for example a
common measurement we make in well logs
is a measurement of what we call density
density we measure what we want in grams
per centimeter cubed oh gosh a fraction
and an exponent right so we found 17
different ways that this can actually be
spelled
so that's challenge one you need to
measure systems lots of them messy
challenge two
data formats this is my favorite slide
so this is a very nice description at
the bottom here from one of the
standarding governing bodies for the
sort of geology subservice industry
and it's giving this statement you know
we've developed new capabilities and
data acquisition we need to offer more
flexibility the result is that we've
created a more efficient and Powerful
standard for log information recording
and we are going to call it the digital
log interchange standard or d-list
okay but
anyone know who this guy is
this is the first George Bush
welcome to the year 1991 that's when
this was written
and they're not talking about some cloud
ready format or Json or XML or anything
like this right
they're talking about a format in binary
optimized for
you're gonna know what's coming next
tape
sweet so we don't have physical tapes
anymore but we have this d-list format
which has essentially been extracted
from these tapes
okay so we have literally data in
ancient formats that we have to handle
next challenge number three go something
like this I know this is supposed to
scare you
Davenport's law of common information
the more important in entity is to a
business well in this case an entire
industry but the more important an
entity is to a business the harder it is
to agree upon the name for it
something you guys have probably also
experienced
now this tree chart is showing the
number of ways we've found to name
remember that measurement of density
this is a number of different ways we
found that this Well Log is called it's
something called bulk density
the proper name the standard is rh-o-b
row b row being the Greek letter rho B
standing for bulk
and it gives us the density to rock it's
great for example Fern Sandstone
Sandstone has a bulk density between 2
and 2.6 grams per centimeter cubed other
rocks are different
it's not purely diagnostic we have to
combine it with other data but it tells
us something about the type of rock that
we're looking at
now bulk density measurements were
introduced in the 1950s
so we have 75 years of Legacy attached
to this measurement
and of course there have been multiple
vendors slightly different technology
Evolution over time we've come up with a
just wide variety of names to call the
same thing and for example in Brazil
it's actually often found with a V
instead of a b because the word for bulk
in Portuguese begins with AV so we also
have language that creeps its way into
this
now the end goal is all of these
different names need to be mapped to
that one single standard rhob
but there's more
remember our unit of measure system for
this particular measurement grams per
centimeter cubed yeah right
so we also have 17 different unit
measure names that these can be in which
gives us a
unique combination possibility of
3910 and this is for one measurement
type we have 49 others
crap
today we've mapped about 200 000 of
these names I'll give you a little more
details of how we do that
okay I'm going to stop here because
you're probably thinking well this is
really crappy why are you going to do
any of this right this is a lot of work
but the answer is that our course our
data science initiatives are also
growing we have a predictive business we
want to be better at prediction so data
science is a natural Avenue to go
and of course we also want more
analytics we want better data
visualization we want people to be
integrating data from different
disciplines better you know so it goes
just beyond it even goes beyond machine
learning and data science but really
about making sure that data is more
easily accessed and usable
Okay so
main problem though
our Well Log data is a disaster right
it's pretty much unusable in any of this
aspects
and in summary the three main challenges
right we got wild data formats a lot of
things optimized for tape and binary
we've got crazy unit measuring systems
and lots of spelling differences and the
naming of our different measurements and
naming of different wall logs have
thousands of unique permutations
now
additionally we also had some equinor
proprietary Transformations we wanted
that in here I'm not going to talk about
that today
but
the next part of the presentation
I'm going to talk about how we actually
solved some of these but remember there
was four challenges I've only mentioned
three
there's one last one that kind of wraps
all of these together
and it comes down to Performance
and the ability to make any of these
Transformations
specifically converting and handling
unit of measure systems and number two
being able to transform with some of the
special Ecuador things we want to do to
this data
okay
to illustrate Challenge number four
we're gonna go for a little walk a
mental Walk we're not actually gonna go
for a walk little sightseeing tour of
Oslo I'm going to start our walk from
the Astra fronting Museum in aqua briga
it's a beautiful Contemporary Arts
Museum if you haven't had a chance to
just walk over Dr brega I highly
recommend it it's beautiful architecture
now we're going to walk North we're
gonna go past the Royal Palace past the
castle and up through some of the nice
residential neighborhoods in Oslo and
we're going to find ourselves at ulaval
Stadium which is the home of the
National Football pitch for Norway
this is about a five kilometer walk yeah
okay I think we can all probably walk
that
um
and this would be about the same length
that we acquire while log data for in a
while five kilometers okay not so bad
right
but on your walk you will need to take a
sample every 15 centimeters
and that is the density that we take
while log samples in a well
so you can do the math
we needed to find a way to read convert
and do Transformations simultaneously
over about 300 000 rows of data per well
if you multiply that by the number of
Wells that we're looking at 120 000 you
are looking at a way of working with
180 billion samples
that's the end member right okay I know
that's the end member I'm being I'm
being quite bold but even on just the
Norwegian continental shelf right we're
talking about 300 million samples
and
here's the kicker
this has to be cheap to process
we're in a business asset we cannot be
spending money on something that is yeah
we have to be very cost control
and talk a little bit about that so it's
got to be cheap
okay so yeah we had some problems
um not 99 problems I'm sorry for the
dated reference
um but more like 200 000 and one
fairly extreme unstandardized data yeah
check okay but also lots of it double
check
so what did we do to try to solve these
challenges now they've been some great
talks already NDC about the power of a
good team right and that was really what
we wanted to begin with
so we got our really first subsurface
data engineering team together
and consisting of cloud architecture
data Engineers some folks with data
management background folks with data
science background and also remember
that person doing that manual process
with that data and that software oh yeah
they're in this team too
and adding on to it the agile pieces
right a product owner and a fantastic
team facilitator who's actually in the
audience so he knows I'm not lying
and importantly this team add ownership
to these challenges right a lot of trust
in all of our skill sets to find the
right Solutions and trust and fail on
things while we failed on so much stuff
right and we had a lot of autonomy which
was awesome
but the important thing the important
point being is that the team was working
day to day in combination across people
who knew the business
knew the data
and knew what also the potential new
uses of that data were meaning the data
science side
working hand in hand with those who
could translate those ideas into what we
could do with cloud services and Python
and development right awesome brilliant
okay
let's go deal with those tapes
for this we actually contracted an
independent small software
um developer team in Ecuador who was
really good at developing open source
packages we decided this we want to go
open source with this shouldn't limit
anybody being able to read this type of
file format
and the best part of this is that the
main data engineering team that I just
spoke about right they could quickly
test and give feedback to that smaller
team creating the open source package
simultaneously right it works really
well plus that small team got the
community engagement as they put things
into Alpha into beta right it's a small
community I will admit that
we also reuse from the open source
Community who had some other parsers
that we needed there's about seven
different formats that while log data
can come in two or binary five are not
binary so we're able to reuse also some
other packages that others have put out
so with our set of parsers we chose
ultimately to crack everything out
into parquet we knew we were going to be
doing a lot of simultaneous processing
due the density of data we went with
parquet
okay
where to start is perhaps one of the
things that gets teams really spinning
and Confused this is a complex problem
spanning
expanding the world literally where do
you begin
any guess what the years are up on the
screen for these countries
these are the years when the first oil
or gas Discovery was made in that
country and essentially the birth of the
energy industry began
now I don't know but my therapist tells
me the more baggage we have
generally the harder it is to deal with
it
and if you look at Norway
1969
relatively quite late actually quite
quite late arrival to the game
so we can make a very good assumption
that actually by narrowing scope to
start in Norway it makes sense on other
reasons of course too but from a data
perspective
we're closer to the arrival of computers
one okay that should help us there
should be some pre-established standards
from these other people who have come
before at least have worked out some
things themselves okay too and also it's
a relatively short time for tons of
derivations to be created in those files
okay
uh of course obviously we were in Norway
so yeah it made sense here to start
and what we found is that by solving the
majority of our unit of measure issues
naming issues by solving the majority of
them here in Norway we were actually
able to take about 60 to 70 percent of
that to other countries
all right so I want to dive a little bit
deeper into this
because to just focus on Norway as a
country was not good enough the
complexity was still too high
so we needed to use the crappy data we
had or the raw data we have on
standardized data it's valuable let's be
honest but we need to use the
uncenteredized data itself to indicate
to us where to focus our efforts as a
team
and it came down to actually creating
inventories
along with our parsers
I asked many many times
how large is this particular how large
statistically large tell me the stats
how large is this particular unit of
measure or naming problem don't just
give me a feeling give me the evidence
and that's what this graph is showing
and this is just from Power bi right
we're just using basic dashboarding to
illustrate some of these and what you're
seeing are those 288 the tree chart you
guys saw later or earlier those are the
the names along the bottom on the x-axis
and on the y-axis is the occurrence of
that name in our files
so makes sense right to actually start
and say okay these names these top 10
names let's make sure we can map these
to a standard and we start here
but you'll also notice that the color
bars on this or indicating the different
unit of measure systems
in these files and you see that that is
actually the bigger issue to solve first
because the most highly occurring naming
conventions
will always have
a variety of unit measure systems right
that was the biggest problem was really
everything on the side on the left side
of that line
now we would do this for every new
country we moved into how
how Wild essentially is the UN
standardization here and let's visualize
that
now this is also important because the
sort of knowledge of these names and
unit measure systems were often with the
teams we're familiar with that country's
particular data so it was much easier to
work with them to show them okay these
are the top 30 we see in your project
can you map these to a standard for us
if you had given them a list of 10 000
right they would have looked new and
said no way go away like leave us alone
right
all right
another solution for us is really
iteration over technology I would be
honest and call this yeah also failing
at technology and trying something
different but you can put it in both
ways
and we really started with some small
python code we had those built-in
mapping lists right that would took all
those 288 names and mapped them to our
friend our rhob
and we could run it on just one small
sector of Norway
we would run I've heard this multiple
times today actually we would run it on
someone's laptop
right and it would take a couple weeks
or so to process about a thousand Wells
the business was super excited about
this though because that was longer
remember it one to two weeks for one
well so they're quite excited about this
say hey this is awesome but still
they're like this is still too long you
can't leave something running for three
weeks on somebody's laptop right that's
just not practical but it gave us the
emphasis to move okay let's move this
over to Cloud
so simultaneously by moving this sort of
uh python pipeline you can call it at
this point two uh to Azure we also
started moving to other countries
outside Norway that meant we had to
separate those mappings of sort of
standardization out of the code right
and we needed to get them into GitHub so
we could probably Version Control them
and also set up these dashboards on top
and the python code itself was then
packaged up to run on Azure machine
using kubernetes and this was a local
equinor deployment of kubernetes but we
struggled due to the density of data
it was hard to know where and in which
file and which sample and in which
transformation step that the data errors
were coming up and we struggled with
restarting workers after airing out
additionally we had little cost control
over the runtime expenses which on the
business side as I mentioned would be
constantly Vigilant over
so we decided to work with Microsoft and
with support from their Cloud expertise
we were able to deploy Azure durable
functions for python while it was still
in preview
and that was awesome it gave us full
control how we split up the processing
to different machines and we combined it
with the applications Insight Services
which is awesome for logging errors so
we could understand exactly if we ran
into a problem where it was coming from
and so this helped us automatically
handle derivations which gave us the
ability to actually code and you know
edge cases as we found them and really
improve the data quality
now today our this whole process and I'm
going to show you in a moment is run on
a zero data Factory pipeline it's an
on-demand we're not dealing with
real-time data here so we just run it
moving into and I have a special guest
to give some details on this in a moment
and we are continuing to iterate over
technology so for example we're going to
be getting to move this pipeline onto
the Microsoft energy data services
but the point is start small and I've
seen this from many of you guys today
start small and iterate and iterate
iterate you're going to fail something's
not gonna work what's the next step
all right
I want to dive now into the actual data
engineering pipeline we have set up on
Azure for handling both unit of measure
naming conventions
but we need to talk a little bit about
well running this Pipeline with well log
files it's a bit like a box of
chocolates you guys know the quote right
from Forrest Gump you never know what
you're going to get
I know I'm sorry but it fits really well
we're dealing with highly variable input
data we're not under any extreme time
pressure right like I said we're not
dealing with a real-time situation here
we're not under any pressure for time in
processing
so the team came up with a very clever
concept to deal with the fact that some
wells just were way worse than others
some of them were way more
unstandardized than others meaning that
some of them just required a lot more
processing than than some of the other
ones
so like the box of chocolates right you
open it up you say oh man this dark
chocolate looking one I bet this is a
really good chocolate truffle right and
then you actually put it in your mouth
and it's like the super sticky hard
caramel that takes you 20 minutes to
chew right we don't know what we're
gonna get when something comes into this
pipeline so we have to be quite clever
so we have highly variable dense data
with need for multiple Transformations
and that's essentially what our pipeline
does today
so I promise a special guest and I'm
going to hand it over to the data
engineer from the team Magnus Carlson to
actually describe today's Azure data
pipeline
thank you Ashley
hi NDC my name is Magnus I work as a
data engineer in Equinox
I hope you all enjoyed the conference
and get boosted with energy and ideas
for your projects
as Ashley explained
we do have a lot of well log files in
different formats how to process them
we use Azure data Lake Azure data
Factory together with Azure durable
functions to achieve this
because of outer scaling of these
resources it will be cost effective
in this data Factory pipeline
we run multiple Wells at once
first we make all raw files into one
common and reusable format with a
combination of Json and parquet files
all of this happens in the log
extraction step that starts an
asynchronously running durable function
after this we continuously pull the
durable function status
until it completes
then we get the information from the
durable from a durable entity that I
gathered useful folder and well
information from the log extraction step
at last we kick off the composite
creation durable function that will
produce our final output
our composite well logs for the
processed webs
this is all for me
back to you Ashley
yeah so thanks Magnus it's really nice
for him to do this for me because I'm
not going to try to explain everything
in data engineering again I'm a
geologist but
the final products the important part
right the final products include the
single parquet file which contain the
master the composite Well Log right it
also contains a corresponding Json file
which documents every step that went
into the creation of that Master file
awesome now we have full lineage we know
if we've done a unit measure conversion
for example right
and we also produce a file that is in
the correct format for our
older oil and gas softwares
so we also have ability at this point to
run a quality check against the
previously man-made Master files to get
an understanding of quantification of
how good are we in our engineering
pipeline against what a human does in
one to two weeks
it may not look like much in terms of a
data pipeline but I think it's beautiful
we've been able to take a ton of
complexity and boil it down to this
right and to me that is quite impressive
oh yeah okay so what do the results look
like now each of these orange dots
represents a well each one of these is
about 35 000 that we actually care about
right now in Ecuador we're able to
produce one of these standardized Master
Composites for and a few more numbers
for you
we dig through fairly unorganized loose
files in our Azure data Lake we dig
through about 17 million files looking
for these particular well logs data so
that represents about a 485 percent
reduction in the amount of data digging
for example a data scientist would have
to do
and the pipeline handles that for us
right
we also receive about a thousand or so
new well data sets per year globally so
we can quickly on demand run this
pipeline right and produce those
standardized outputs and of course our
machine learning Engineers or data
scientists can quickly put that right in
because it's in the same schema huh
great
plus if we decide we do need to make a
schema change to either naming
conventions or a unit of measure systems
there's only one place we have to do it
right which is in this Pipeline and we
can rerun all of the outputs
but the capability to mix and match the
data from certain countries right for
different rock types different types of
rocks in different places in the world
and different situations it's an ml
training data dream right this is a lot
of really good statistical data
also and we we publish actually papers
on some of our machine learning models
on this but we've supplemented
statistics from one part of the world
to be able to support a project or
prediction case in for example Norway
right where sometimes we need a little
bit more boost we have similar rocks for
example in Australia we can take the
files from Australia to help support a
machine learning model that we're
running for an application in Norway
additionally as I mentioned this works
very well because no matter where we
work in the world we have access to the
exact same data
now the pipeline costs about four
Norwegian kroner for well it's about 40
cents which is extremely cheap if you
remember how much we pay between a
thousand and three thousand dollars
and if we rerun anything it's about 140
000 knock which is approximate price we
pay today for just six walls
and it's pretty darn fast right it's
about three minutes on average per well
and that's though quite a lot of
standard deviation on that because we do
have some files that take a really long
time to process but on average about
three minutes
okay but it's not just about numbers I
think I really want to mention that too
this was really a first foray for us
into Data engineering on subsurface data
and it's affected a couple other things
right
um there's a big learning process so
it's also affected how we look at our
data vendors and our contracts right
data comes for us comes from someone
someone goes and collects it and we
usually hire and pay for that service
right and so for us it tells us okay
there's a lot of value ensuring that our
vendors are using right standardization
that the formats they're providing to us
are machine readable and properly filled
in with right metadata with high quality
right and they're all using the same
types of units and measure systems right
additionally this is one of the first
teams that we had ever assembled to do
this type of you know cross domain data
engineering we have the business people
and sort of your your computer
scientists working together and that was
really brilliant the mixture of the
business knowledge of the data but also
Cloud systems architecture apis cloud
services all these things
combined with how data science wants to
work so it's that mixture of the
traditional the new and the sort of yeah
computer science uh knowledge that
really made us capable of handling all
of those challenges
finally I'm just one last number I know
I couldn't really like numbers but value
realization
I need to mention that automation is a
pretty scary word in a traditional
business
and starting with a simple thing a
seemingly simple thing like this
standardization of data is something
that people take a little bit easier it
doesn't say oh we're not I'm not going
to lose my job here you're actually
automating the really crappy things that
I don't want to deal with right
so this was a really big sort of input
and sort of the culture that sort of
automating and working with things that
are maybe not so scary is one way to
really introduce new technology into a
more traditional company
and in our case we also demonstrate a
massive upside if you were going to
produce these Master Composites for
everyone of our wells to prepare for
some sort of data science application it
would take
253 manures basically we would never do
it right
all right I'm almost done
I hope over the past hour ish 52 minutes
maybe I've changed your mind a little
bit as to how Ecuador is approaching
data and software
yes I still have to do a deal with sap I
actually did my time writing this
morning in it and it sucks as usual but
my days have a pretty equal balance now
of the Azure portal which makes me feel
a little bit cooler as a geologist
now
where do we go from here
I'd like to say
put a ribbon on it pack it up we're done
but in truth we have a very long way to
go
in subsurface we have a massive data
footprint and the new energy chains that
we're working with offshore wind and
carbon capture they want to reuse the
Decades of data that we have in-house
right
so I've told you the story now for just
one of these data types there's a lot
more to go
so the main point being that we need to
continue with good partners and good
teams right both computer scientists and
I.T professionals like many of you in
this room working together with folks
like me
so thanks for listening today if you
want to check out some of Ecuador's open
data sets including things about rocks
and subsurface you can find that at
data.ecuador.com and you can also check
out our open source software offerings
thanks all you can catch me at the
equinor booth if you have any questions
so
[Applause]