good morning everyone
welcome
I'm going to talk about how we at tomra
work through
the chip shortcut that is that hopefully
had happened but
it's still happening
probably just a new normal I get
yeah for those of us that don't know
what tomra is we make reverse vending
machines in Norwegian is called pump the
machine
what we essentially do
we take these bottles
and then you get money back
it's the whole concept was
the the stores the grocery stores in
Norway here in Norway they have a
responsibility to accept back
all the empty drinking containers
so have no way we
when we buy one of the one of these
hopefully with something in it
you also pay a deposit
and when you return these to the stores
you get that deposit back
it's
the whole goal is to prevent a lot of
littering and to get all the material
back in control fashion and not just
laying around in nature
so the concept was the stores found it
to be too much manual work
sometimes people came with like garbage
cans garbage bags
full of like
beer bottles 50 years ago beer bottles
was glass
and I was quite heavy
and all this
grocery stores wanted
something to automate this process
so
our Founders 50 years ago
the Planck Brothers they invented the
hole in the wall
where essentially you go to the machine
you place your bottles in the hole in
the wall and then you get the receipt
back with
how much money you are owed from the
grocery store and then you go to the
checkout and you get that money back
so here we see one of those customers
he's a very good boy
not all of our customers are such good
boys but
hopefully someday
so we have quite a lot of different kind
of machines and in order to have a large
configuration of machines we have
decided to build them up from
common modules more some more common
than others of course
so examples are like we have a component
that handles a specific stepper motor or
a bldc motor we have some i o expander
boards we have an in-feed like we have
belts that when you put in a container
it feeds through the machines and stuff
we have power box we have miscellaneous
lead displays
so we get all these kind of machines
so we have from
small machines that are completely
standalone
to larger ones these are one of our
newest
babies
we like very much
where the concept is you open a big
hatch you take your big bag of empty
containers like bowls and cans and it
just pour it in close the lid and just
wait until it's counted up for you so
you don't have to manually do one by one
by one very helpful and the customer
seems to be very happy
when my both my mother
was happy that they got one in their
town and my grandmother was very happy
and like okay this is something that
people start to notice
but these kind of things has been
very common in the Norwegian stores in
yeah like the last 40 years perhaps
uh
but we also due to having these kind of
modules have started to create
like huge centers that are more for uh
trained Personnel that can count huge
amounts
like thousands and thousands of bottles
and sort them correctly say this one
goes back to Coca-Cola this goes to
shredding this one is glass and that
goes back to
destruction and everything
so we have quite a big array
they're all from like this huge to
configurable size backgrounds we call
these ones back rooms because we have
the front end in the front and you have
a back room in the back like before the
wall and back of the wall
some of these can be pretty large uh the
big one up top right and the middle they
can be like nine cabinets long
and then that's that's very very large
grocery stores that handles huge volumes
an example of this is like on the Norway
Sweden border there is quite a lot of
shopping for
let's call them cheap
sodas and stuff and some of these
machines with like the huge its
background they go
as long all day like from those moment
the store opens people are feeding them
containers and they just work well
like thousands and thousands of bottles
each day
so yeah we we usually speed it up in the
front and then back end you know back
room thingy the front and usually
handles like recognition life figuring
out this is actually a bottle and not a
teddy bear
or
something else
we had eggs and sandals and chocolate
bars and everything through our machines
when we tested the big one that was
weird we got like other we got scissors
we got yeah a shoe we got a half a teddy
bear I think
a hard-boiled egg that was fun
so we got some fun challenges when you
say hey here's your bag just throw it in
then like
there will be some weirdness up in
the front just recognizes and says this
is okay this is not okay figure it out
how much to refund
handle all the people that want to cheat
like attaching like a fishing line
on the end and as pointing in just
yanking it out don't worry we detect it
you don't need to try
these kinds of things
and then in the in the back the
backgrounds are
responsible for like sorting the
doing the aluminum cans to one compactor
and the pet poles like this to another
and when one of the bins Go full you go
to the next one those kind of things
so we make things actual physical things
and short business lessons if we cannot
make things the income stops and that's
bad
do I get an MBA now
okay
business has assaulting get money more
in more money in than out
so like
one and a half years ago things started
to get
hard to
find a supply for
so not only this this that we are we
many of us have heard about like chips
and electrical components are hard to
get but also metal
got harder to get
price here
and like physical buttons maybe you can
plastic was also kind of hard to get for
a while and a lot of these things you
can easily find replacements for like a
metal sheet a sheet of metal is
more or less exchangeable with another
sheet of metal
but replacements for microcontrollers
are trickier because they are more
yeah they do more stuff and they are
intelligent beings not not beings yet
but you know
so
currently we need about a hundred
thousand microservice a year
roughly
and
when we have those kind of volumes we
have to coordinate with suppliers very
closely to have
I guarantee that we get the supply
when we need it and we have enough so
that we can create the machines that the
stores are waiting for
but as the supply gets worse the price
usually goes up
but there comes a point when the supply
actually stops
and then it doesn't matter how much it
costs because you can't get any
so the price essentially goes to
Infinity
so
then you're in a pretty bad place
because you cannot actually physically
get the component at all you can't even
buy your way out
so away through
I say we start to get really worried
about one and a half years ago in the
winter of 2021 January February those
around there
and then we they started uh to really
look for Alternatives find other
suppliers for a bunch of stuff
uh tried all they could to keep
engineering out of the decision
all praise to them
they do fantastic working in the
sourcing and supply
but then it came a point where they say
this is we are too worried about getting
enough components especially the
microchips the prices are doubling
they are tripling they are
five doubling they are 10 times the
price suddenly they get they got
extremely extremely expensive
so then the decision was made that
this is untenable we need a different
strategy we need to fix this in another
way can't buy ourselves out
So the plan more or less was you find a
different microcontroller
so
but the thing is you have to find one
that similar enough
and then you get
you think about the capabilities you
need and stuff
so that's all well and good but then you
have to design
that microcontroller onto the existing
circuit boards and all the panels are
always different like generally speaking
they are different pin compatible is
usually just pin compatible on paper
there are always some greblins there's
this
just a reality I guess
so we design
copies of the current parts we duplicate
everything very strictly
and then we have to change the code so
that it supports both the current and
the new one
at the same time because then we have an
alternative for the sourcing and the
manufacturing of the price
and then the interesting one we have to
make the new models work just like the
old ones
and then that's
as always
the tricky one or another tricky thing
so as I said we had hundred thousand
microcontrollers but there are not all
on the same module
so we do the most needed modules first
here is roughly the distribution of our
microcontrollers and the different ports
not labeled because
business sensitive I guess
so some of the modules are used by many
different things
and those are the one on the far left on
the screen here and
the idea is to just work your way left
to right
take the one you have the most need for
on uses most users this should be fairly
expand fairly straightforwards
but a consequence of this is that the
development for the next part reduces
always when you go downwards because you
have if you start with the one on the
left you have a lot of things you have
to change in order to make it work but
then you can take some of it over to the
next next one because they probably use
this sort of the same thing like the it
uses the uart it has a command line it
has per referrals controlling and then
some boards have analog the digital
converters that they use and then if you
use if you implement that far left and
then you need it on the more to the
right then it's free
essentially
so the one on the for this left is
this one
I think it's this one we have now three
of these they look the same you can
exchange them in the machines just take
an old one out put a new one in and it
still works
so this is
a circuit board with controlling a motor
essentially and it's just a stepper
assembly a stepper module for your wrap
in the Box
and then we can use that in different
kind of back rooms different kind of
machines
and some of the things it does is it
communicates over can to the main
machine the main front-end the
controlling part of the entire system
the primary purpose is the motor with
positioning uh absolute positioning
but it also has a few digital inputs and
outputs and an analog input so having it
be just a motor would be would be fine
but having a few extra inputs and
outputs always helps and you can
suddenly use it for like a button or you
can use it to flash some lights to guide
the attentions of the
stuff in the stores and so here
so the usage of this is like roughly a
third of all our microcontrollers that
we need are used in this this module
so here it is attached to a flat bond in
the back room and we have
buttons going on the conveyor here and
then this flap pair with guides some
objects some
bottles into this chamber here which
then would drop them down in a compactor
which
crunches the bottle down to something
more manageable size so it doesn't so it
isn't just air like a lot of this is
just air
we can use it to use it for other things
we have one of the things we have is
for primarily for heavy glass bottles
you don't want to just drop them in a
bin then it probably
uh shutters
but we have some that can carefully
place them down on the table and then
it's you have it standing up and part of
the mechanism for putting the guiding
the bottle of the conveyor and down on
the on the table have a motor motor that
kind of half grabs the bottle so that
you need the positioning and this
stepper module used both places
perfectly fine
so
we we had this one and we needed a copy
uh we needed a copy fast
but it has to work just like before
right okay this is kind of tricky it
shouldn't be but as always you probably
will hit all the shortcuts you have
already taken
like you skipped some documentation you
skipped some features you
you have some implicit knowledge in the
guy who left the guy who or the person
who left sorry
so what we are
kind of on the deadline right
because if he fail now
we basically cannot fail because if we
did if we cannot create one of these
then the production stops
and at best we have to put all our
assemblers on the under administrative
leave at worst we have to fire them and
that's not good
so we have to
go steady we have to
be pretty sure we'll make it actually
replaceable
so initial knowledge is like this it's
used in several machines and back rooms
perfectly fine we do have production
checkouts and all these physical
dimensions and everything we have
documented the protocol that it
communicates on on canvas it's
proprietary but we still have like the
interface description and everything
it's well documented we have some
documented Behavior
uh because usually you say that the
requirements are for complete machine
you don't always bother to do full
specifications and requirements for
individual modules of the big emergency
system
so we need something slightly more rigid
more proper more maintainable more
documented we need basically we need
more confidence that when we have to do
something we doesn't want we keep it
keep the current working as it always
have and the new ones working like the
current
so we the voices itself some safety nets
we create the safety nets hopefully the
race are document knowledge and verified
that it still works on all complete
machines
primarily we do this with tests we have
at low level Zone on the top of course
unit tests we have model tests we have
system tests and we have Market test
so
this is probably not controversial but
module test can also be argued to be
some kind of integration tests but it's
when you start to use these words you
get slightly fuzzy
and people use slightly different
meaning for the same words
we can't even figure out what a unit
means anyone have
a unit can be like tiny or it could be
big or you can even call this a unit if
you want
the things I get well
it'd be what it'd be
so we try to set ourselves up for
success
and have these kind of safety nets more
layers of testing it
so we cannot of course rule out balls
but we hopefully catch a lot of them the
system tests are
pretty okay we have a bunch of machines
in the labs they are mostly manual tests
like you have an actual tester we are
lucky to have a separate Department that
do testing and verification
uh
mostly manual tests like they set up in
the machine in some particular way if
you did some bottles see that it works
or like the joint the doors at the wrong
time I see that it falls back and it
rejects the items you want and that it
responds to errors and a lot of these
things you can check you easily check
like a lot of the common configurations
out in the markets you can also check
some uncommon configurations
usually they use like hot built
or Alpha release software so you can
give me the software that this on Main
now and let's test that out
so we have in total like we have a lot
of machines but then we took some of
them and dedicated it to this project
Market tests are when you go out in the
fields you agree with some store owners
that hey can we borrow your machine for
like a month and they say uh okay kinda
and say oh yeah it will
will give you some slightly better
service or
something but it's a someone else not me
agrees with the Stars that these are
okay
we expected to run Bay
a separate release for these kind of
tests are based or released usually
and for this one we only used the
refactored firmware for the new uh
modules because then we know that we
cannot break the existing models modules
yet
we when we started we don't have that
good confidence but we build constant
confidence later and then we can flip
the current one
on the in the delivered software do you
see everything new
so these kind of Market tests are like
usually a month long uh we instruct the
stores to just use the machine as normal
and then our people tested and Technical
products managers will monitor the
machines
through an online service that we have
back office system
so we have things like measurements or
error situations and figuring out what
the current state is and
gives us logs when everything goes bad
and such
so unit tests sadly there were none
like you said in terms of the test we
have no tests
which was kind of just the reality for a
kind of Legacy system
to say that the firmware and these kind
of things is probably like 10 years of
history and these kind of things in
different modules so the initial
architecture didn't really support
having unit tests so there weren't any
and then you update the code and then
you cannot add and then they just stayed
that way
but we do have some now so
if since we cannot just add the unit
test we need some kind of safety net
around the thing so we go one level up
and reduce module tests
and these are our pride and joy this is
the primary reason for this talk is we
have
these kind of racks
uh these are normal server racks but we
have
both in text modules in there and we
have Standalone circuit boards in there
and we now have like a thousand tests
that runs
and just checks actual physical
components for some desired Behavior
so we run this manually or we can run
them nightly we run them incrementally
on each new checking of the source code
and we build them also on reviews
so basically my pride and joy our pride
and joy so it's kind of like black box
testing for the complete modules so
these kind of things run in like 10
minutes for the review builds or the
incremental I think
so it's fairly fast enough I think
but when we have these kind of module
tests we can now refactor the code and
then add unit tests as well like because
we have the safety net we can refactor
and this you sure that we probably catch
a lot of the things in here
so the contents of the rack is like we
have a PC we have a network switch we
have some serial to ethernet laser
terminal Port converters
we have power converters to deliver 24
volts to the boards
we have power switches on relay so that
we can physically anchor the power to
all the things
we have USB to cam adapters so that we
can talk to the boards on can from the
PC
we have pullout trays like below so you
can see
and then you some we also have some
webcams
so let me see that the boards actually
physically move if we want to but it's
primarily used while you're developing
the test and you see that oh yeah it
blinks it blinks with the right
speed and kind of things very useful
when you are sitting at home but
everything here is in the office
so
you're sitting at
your desk with your fan all your
monitors instead of it's okay but but
all this is in a lab somewhere
so we started with one cabinet we're
currently about five and it wouldn't
surprise me if we doubled up by the
summer next year
uh we continually just add more tests
for the current modules we add more
modules because you have to start adding
the modules we haven't
decided to replace yet because that's
the whole thing we we started saying we
have to replace like one then it was
three or four then it was six and by
last week it was eight different kind of
modules so it's a new normal that new
models would always come
we're also in the process of adding jpeg
the connected modules those will be
exploded modules with the Standalone
Sports so that we can more automatically
test like boot loaders and fault the
upgrades and stuff
but our primary use at first which is we
have the module axis and then you update
it over the normal updates over account
as the reverse vending machine would
have done itself
so yeah we started with one cabinet it
was like let's just try this out the
engineering more or less forced its way
through they we have to have this
and
yeah the unforeseen benefits are let's
say I'm not forcing to some but we do
now have a very detailed behavioral
description of each module in this test
this test track
and it's just not for the
developers but it also for the system
designers
so they can now know that this one has a
quirk that if you calibrated to within
plus minus like 80 90 degrees then it
stores its calibration position but if
you don't
then it loses its calibration calibrated
position when you reboot the nodes so
these kind of there are some funky
things
that all have that all the engineering
or the developers just know because of
course it's always been that way but
when you have some new system designers
they cannot find the documentation then
you can't it's not feasible to just run
around and ask the entire department
so in Century we get this back we always
should have had that was the shortcut
that we took so now we have to backtrack
on that
in a sense we have an executable spec
which is kind of fun
in a sense
we we reuse the can implementation you
know the logic for of upgrading and
downgrading and
flashing and communicating over the
proprietary protocol on can
from our reverse vending machines the
actual software which
if happily enough found bugs in our
actual
other parts of the system
and then we had to improve the
architecture because we have to slice
off
the kind of implicit dependencies that
they had on like it knows whether it is
what kind of machines it was but now it
is it isn't a vending machine it is just
a Standalone thing so it improved the
architecture this isolated quite a bit
more of the actual implementation for
this
and a fun side effect is we bought the
cabinets that had a lock on it Key and
Lock
because we think that's usually good to
have something we have to move these to
a place where they
a lot of people can reach them but it
was mostly helpful because someone some
Engineers just borrowed a module from
the from the test track like all the
tests failed and like okay this is bad
and we just locked
locked the doors and placed the key on
one of our developers desks like
everyone knew it was there but some you
have to like physically go get it in
order to steal it and like that's just a
little bit of an hindrance and then that
stopped
but now everybody's learned to know
they're all open all the time
so
this module tests are used by our review
system
we do one of the things we decided to do
was to review every change related to
this activity like everything
because we we don't want to lose that
common knowledge in the teams so we have
to want everyone to have a look at it so
the review is built for all different
kind of modules it's
yeah every module you run all the unit
tests the few we had hopefully
increasing then we've run module tests
in the rack so this takes like 10
minutes but
that's fast enough because you can do
what I call CR driven development
like you have you develop on one or two
boards and then see oh it probably works
create a review don't send it to anyone
yet but then you just wait for the
results because then it's run on like a
thousand tests on 20 boards in different
configurations and stuff and then after
with that you can do the normal review
behavior and see at we're probably good
on this check this commit
so this helps us catch bugs even before
they are committed into the source
so this has all giving us what I like to
think of as confidence and velocity we
do have a very much increased confidence
in whatever it is we do the conference
in the code and everything but it also
gives us velocity because like we we
know the direction we're going and this
guides Us in that direction
so if my personal belief is like these
kind of directs we created it's like one
of the most important development things
we've done in the last five years
so that's the high level and then
probably well so many people like to
know some details internally
everything is C plus plus
with another slight footnote it just
started in C of course as everything
um
but we are moving to make everything Z
plus I don't know
so this the technology we try to keep
things boring with little less than
dependencies we have an in-house
protocol that wraps count
uh and then we use
pre-orders and we use some link time uh
abstractions for some functions and we
use after bake classes for like
C plus plus each things we decided on
Virtual functions because they're very
easy to teach everyone knows them and
they are usually fast enough
if they're not fast enough you have
measurements and then you do something
else so yes
uh we haven't hit that yet so fast
enough
uh we rewrote the oldest build system
from uh can you make into cmake because
we use CMA for the other parts of the
system and let's just focus on one
technology
so we are kind of uh we worry a lot I
think with external dependencies and
stuff but
I'm still unsure if that's a good thing
or a bad thing
so
but we try to use like known tools
boring tools
techniques that everyone knows
uh part of this is also we acknowledge
that there are Engineers know different
kind of things they have different
skills they have different levels of
proficiency and CA plus plus at least
so we try to keep things
as far away from the quiz last night as
possible so
the high level is like most modules
we've decided to use the same binary
image in Flash
primarily because we do both can do bulk
loading over can
McCann is not a particularly fast bus
and when you have to transfer like 100k
it's it is a significant speed of when
you have like 50 nodes that you have to
update
uh the downside of this is of course
that we have to do a runtime switch on
which kind of module we are
so this kind of board type is persisted
in during production checkout it writes
to some
a persistent storage and then we can use
that
at runtime
so another thing is then the image has
to fit on all boards so we are kind of
limited on functionality but it hasn't
hit us yet when when that happens we
probably just give the the
the modules that need all the flash
there they have they will probably get a
separate binary image and then we get
some of the benefits of both loading but
not completely
and one of the things are bootloader
waits for a start on the canvas so the
when the modules come up they just say
hey I'm waiting and then the machine
says hey yo go start
uh the both your boot order can also
flash image in the case of you get
involved the
body image and we do also have Double
Image in Flash just to able to go back
and forth if need be
very rarely do we need it
so the link time platform abstraction is
what Kate Gregory mentioned in her
keynote yesterday it's with that we move
the logic we move the work for into the
build system now we have these kind of
PR functions and then the build system
will add
an implementation for that that's
specific for these the MCU that we're
working on building for sorry
so we try to move away from if that if
deaths and all as much as possible
because it's it gets hard to track when
you have like triple nested and
kind of stuff so of course there are
tons of improvements we could do we just
go
step by step here like IQ number we
could probably be a class same as we
could have a pin that do everything uh
the lead could have been
a separate class and stuff and all the
things we can do now that we have the
tests to catch all our bugs
we also use base classes abstract base
classes usually with pure virtual
functions like this
uh
of course for us we had a lot of
existing classes that do stuff like an
angle sensor and a counter and
and allowed to digital conversion and
stuff uh part of what you do it just
extract an interface from that and you
have these kind of things
and then we'll get to work on
improving this like adding on separate
RPM type instead of just using floats
bunch of things we can do differently of
course
since we have runtime switch on the
module type we have the case that we
have
currently we have three versions of this
as we have this kind of thing in three
different mcus but we want them to have
the common setup we want to guarantee
that it all they all have the and the
digital conversions and stuff
so the pattern we use here is we have a
templated factory function essentially
so these are the three boards for this
stepper assembly we have
the
the code essentially boils down to you
come to a point in main that you say I
know my board type
and then there's a switch and a nice
switch case there is one entry like step
assembly of board 527 create that one
uh
we do have then we have this uh template
type
template argument named board and we
have information from that board like
there are slight differences between
everything because the you find
improvements that you could have and
like you have this kind this is all the
oldest one I think and then the engineer
electrical engineers and stuff they have
a list of like we know this is kind of
we wish we had fixed this this and this
and this and so the the modules aren't
completely the same but they are
interchangeable and put it to a maximum
degree but they also have some extra by
just necessity
so the Hope was that this kind of
function will more easily show you what
the differences are between all the ones
that have
what is the same for all the modules and
what is the differences when you have a
different MCU
so here for example you some have safety
input somehow you have to know which
kind of pins on the chip we have
connected the motor onto some of them
have different places what they put
today I led
now if it's a fun one and that is also a
run time
where we have to figure out with
revision of the boards has the idle at
this position like which kind of rotary
encoder does the board have because that
phone also had to be replaced
uh some argument does it have an
inverted
like I said mounted upside down by some
error or something
nothing so
uh what's fun of this is like we've
we've added this for every new module We
Touch
then we like a week ago or two
I overheard someone to say like we don't
have started on the new model yet but we
will they created this Factory function
like so we have it ready for a lot of
the modules that we know were going to
change
one of the things here uh this sense get
sensor type which kind of rotary encoder
we use
we have not just we not just no but some
of them have uh run time you know it has
a register on the board it says whether
it's the one or the other so then we
have to read a pin in order to figure
out which rotary code we use
and that we can use like a normal
function overloading in C plus to take
either you have that detection pin or
you have or you know uniquely which one
you already have
so you can have this kind of
uh if it chooses which one of these
based on what the structs of the board
actually defines
it's kind of fun and it's it boils down
to nothing essentially because most of
these are can be a everything that is
has empty but it's probably a static
context per variable
so it that one would just evaluate to no
for this one perhaps
some things uh yeah
of course it's slightly redacted to
protect the innocent there
uh one of the fun things we also have is
we have panicking
sometimes you get hard faults you get
where desertion failures and like
when that happens the best thing for her
and MCU for an Abel system you just for
these kinds of lists we don't want to
recover we just want to go go down and
just
send back to the machine hey things went
awry I'm now at the start
get me up to speed again
you can send like the Panic reason and
you get like back Trace what's the
current rtos track task and all the
active interrupts and fault registers
so that one gets logs and it gets sent
if
if it happens in the field and then we
can get all of these things
when the bug is reported like a week
later usually
this is just a section in the Linker
Drift We defined some area of the
memory that these kind of
data resides in why you're doing a soft
reboot
one of the things we use there is like
in macro or that that's built in
unreachable
you can also have functions marked as no
return
and that tells the compiler to like this
one would never return and then would
optimize it slightly better and it
you avoid having to create some return
zero or
something afterwards because the
compiler knows it can whenever you come
to here ever I'm dead after
this kind of code is a snapshot in time
and there are a lot of improvements but
we have to choose our battles I guess
so we are working continuously on making
the base architecture easier to explain
them better so it's only been a year
so we've done this
which is where to think about now
there's only been one year
and we've replaced like six complete
modules now I think
we do seem to tend towards more and more
classes just for isolation and
compartmentalization and better
modularization I guess
it do make it easier to understand
dependencies and that field that's one
of the
big benefits of C plus plus and
hopefully it likely makes it easier to
use in unit tests instead of having to
manually add the function that it
doesn't
that is missing
and you avoid kind of global State as
much as you can
so there is an internal discussion of
whether to use like uh
Parts the specific Factory functions
like create me and ADC under digital
converter or whether we just want to
create that one in Main and pass the
reference around
we'll figure that out I guess probably
will be influenced by the unit test and
my personal stance is that for this kind
of system main is the factory function
that's everything
main is the only factual functions you
need
but we are curious about other projects
like lib Hall which are
primarily authored by Khalil Estelle
well development for his in the
University course where they use free
artos very much like us and that one
provides ready interfaces for pins and
registers and SBI and stuff so we don't
have to reinvent the wheel down we'll
see whether we go for it or not again
it's only been a year
also again confidence and velocity we
build our safety nets in order to give
us confidence and
ease and or peace
we want calm Engineers so
anything you can help them avoid
like having multiple thoughts usually
helps
so now the survival
we do have new modules they work they
have very few bugs
remarkably not remarkably
as expected
so now the production for our machines
have Alternatives we know we are no
longer single single source
and we have like it does cost something
it does comes it's cost to have multiple
sourcing and the multiple production
lines and stuff but
this may be the better way in the world
that we see now
so we now can continue to create reverse
vending machines which is good because
then we get money
and then I get to work still work where
I work which is
good yes it's good I work well moneyp
keeps coming in
business lessons
uh
but it isn't just that we have
the survival continues
we are not done many had hoped we were
done that this kind of shortage will be
over but it's not
only is the components shorting but you
also have shortages on actual Manpower
in Productions in like China
at least in
Shenzhen and wherever where they have
con the the covet epidemic is still
ongoing very much so
but
we continue to survive we have a stable
team we have people that work exclusive
more or less exclusively on this they
get their quiet time they get their
attention on the right place
and but we also have enough people to
cover some distractions so because
distractions will happen so you have to
have enough people to cover some of that
we have significant knowledge sharing
within the department before we had like
two people that worked on this code base
like they only part-time on this
coverage now we have like five people
working on continuously while this
project is ongoing
but now we we are much calmer about what
we can safely know about the system and
do changes in the future
uh
thankfully now management recognizes the
importance of these module tests and
module rack we no longer have to beg on
our knees for a new one we just say we
need a new one and when we get it more
or less the model rack is basically
placed by everyone
uh I love them we love them the models
keep happening
when the deadline for this presentation
was I think we had like four or five and
then before summer when I primarily
worked on this they had seven and last
week we decided to do two more the the
survival continues but it's
it's interesting to note that the
delivery is very predictable
for every module with hike we redesigned
it takes five months five or six months
from when we started to it's uh ready to
be mass produced and then what we're
doing means sourcing and supply has also
had a month or two to figure out all the
supply lines in
but I looked at all of them and they
they offered in like five or six months
and like the development and as I said
earlier the market tests are one month
and if this Source against it probably
has one to two months and then you have
some extra internal system testing like
development is like three months for one
of these things
so
is remarkably stable
the
but yeah the survival isn't over this is
more or less a new normal
but we are ready for that now we know
what to do so we have introduced calm
enthusiasm within the grip
we keep going and everything but we're
still calm we aren't stressed
uh
this isn't survival
it's it's for people it's for us
it's okay we are the actual humans in
the system here we have to be happy we
have to do good work and we have to make
it so that we are thriving
and I listened to this firsthand like
all this safety net that's they make it
easier to talk about to make it easier
to reason about the system and like you
aren't afraid to break anything because
hey brakes happen like every day we have
okay this test started failing nightly
like okay is it a problem with the test
track or is it an actual bug but hey
it's it's everyday thing
and it's I'd like to just say that I
despise the word stress for me it
actually is like
uh if for me it means destruction and if
if you choose to think that stress can
be positive I encourage you to think of
a different word please because stress
to me I associate it with like Bridges
and if a bridge is under stress it
collapses
it you destroy some joints and it is
actually a destructive work
there
go study
and get forwarding yeah
thank you very much
yeah uh do come work with me we are of
course everything every day looking for
other people more co-workers as I think
everyone is uh special thanks for to
work for encouraging me to do this
pushing me
I've been extreme and obviously two days
but it's better now of course thanks to
my friends and also thanks to my mom for
this amazing story I love this
any questions
um
we don't have so many developers that
have needed the question was whether we
have to have time slotting at all to
test our stuff on the rack uh
we do have some parts of the rack that
are reserved for just manual work but we
also have uh when we can disable the
tests and around but we are like five or
six people that work on it so we usually
just talk together
and they arrange that
yeah oh yeah how fast we do board bring
up
um
uh some of these can be pretty fun too
we had some one example of this was that
we had a
a boards that we are ready for and when
from it arrives on the desk of the
developer to it was working it was 30
minutes
and then it mostly works there were of
course some bugs but it was mostly
working and they'd come up and talk
everything because we had uh all the
frame uh the rest of the stuff working
on a different port so it was very
little and if it was this one you
you had all the policies correctly you
had all the parts implemented for the
old way and then as long as you did the
small platform specific things correctly
then it magically appears so like I
think on average is like a day for bring
up for us
[Music]
yes
the question was whether we have
considered using device Trace yeah or
some other way to set
the answer is we haven't gone that far
yet I think we focusing on the cars as
it is and working towards but I can
Envision a future that we try to use
this more constructively or more
declaratively
I thought I'd leave you with some fun
bugs we have found during this
um we had one kind of device that when
you try to use two of the outputs as pvm
it just said no
but we didn't know why so we didn't know
it with the bug or it wasn't but it's on
it but it's we used but it's
because of a hardware timer that was
used was shared between two outputs that
was fun we had the rvm going into an
infinite Loop if module refused a bulk
up thread I was luckily catched in the
tests we were reading wrong memory once
hopefully thankfully it was just where
we print things on version so
that I think that one was caught by our
system test our tester
production checkout failed ones because
we changed the command line output
slightly
once we added Ox before hex hex number
and then it broke well
we're slightly aware about that the fun
one I think was that we had some we have
some uh the belt that are in the front
of the machine that first grabs this
bottle
that had to violate communication link
between the actual boards that run and
the send some docking underneath that
commution dropped after exactly 21
minutes and 24 seconds
every time
and we only called it by looking at the
logs because the log said oh a lot of
communication whether the with the invid
with the device and then it just
restarts and comes up again and then
it's up already in like a second or two
but then you inspected the logs and that
hey I lost it after 21 minutes and 24
seconds at least and then every time
like okay turned out to be uh a
compensation for some
frequency modulation that's just drifted
there was a slightly off by one every 10
milliseconds and then it reached uh yeah
in a short Max I don't think and then
died and then Rebooted
funny kind of things but
still very happy with the work we've
done
question
yes
we do have hardware and software
watchdogs and when the hardware R was or
software Watchdog triggers then it
causes a panic and then we record that
and we communicate that well so we catch
a lot of things these kind of things
Hardware faults and stuff we try to
catch everything
out of time thank you all very much