AI-Assisted Laparoscopic Cholecystectomy: A Live Demonstration

okay so i think that we are connected to the congress in roma i'm bernard almay i'm together

with dietro masconi and it's a quite unique experience because we have connection all over

the world in roma in in barcelona and in the operating theater in strasbourg france at the

university hospital with the professor who is the head of the department of surgery

And we have quite unique opportunity to show the work done by the team of Nicolas Padua in France on artificial intelligence.

And this is a program that we developed some years ago, looking at the potential of using artificial intelligence to guide surgery during laparoscopic cholecystectomy.

type. We are connected to Strasbourg Didier Mitterre, who is in the theater, and good to know that

Didier Mitterre has no access to the image, so the audience in Roma Barcelona is the only one who can

see the machine function. That will not interfere with the surgery of Professor Mitterre. So Pietro?

Well, you did a fantastic introduction. Thank you, Dr. Dallemagne. And I'm super excited to see finally the demonstration of these different artificial intelligence model during clinical procedure.

Again, I would like to stress the point that this is a proof of concept demonstration,

which will not alter the surgical procedure as the surgeon does not receive the feedback

from the analysis of the artificial intelligence.

But now I think I will let the procedure start because that's the clue of our connection

here.

And soon after the procedure, I will have the pleasure to speak a little bit more about

the various artificial intelligence algorithm that the team of professor nicola padua from strasbourg

the university of strasbourg and the institute of image guided surgery issue has developed over the

last years in collaboration with irked now i'll let the i'll let professor bj muter and dr alphonse

la pergola over it we are connected in life dj thank you so it is a case of a patient that has

It has a previous common bile duct stone with an ERCP, biliary stones.

And we started the cholecystectomy.

The ERCP was two months ago, so everything is clear.

I've just started by the introduction of the troca.

I think it's the aim of the day.

And I have made the entire opening of the peritoneum

and the posterior opening of the peritoneum, nothing more.

Now I am going to go to dissect the pedicle.

and now I will let the machine make the work and I don't see anything so for me it's like a very

normal cholecystectomy without any interference in my procedure so it's very safe the patient

is aware that his image is used for teaching so all the all the safety recommendation and

information are done my objective is to do as usually to identify the whole anatomy

people can see basically this analysis and we can just introduce it. So first of

all I would activate the face detection model. This is one of the first

artificial intelligence analysis that have been developed by the team and

and other teams as well,

because basically it is important to understand

the steps of the procedure as a junior surgeon,

a surgeon in training does when learning a new procedures,

also machine need to understand the development

where of the surgery, the status,

where it is and where it's going.

And the main point of this is to gain an understanding

of the context of the surgical procedure,

but there's also some, I would say,

direct and indirect use cases of this face detection.

One could be that the all operating room staff

is aware and ready about the status of the procedures

and ready in case instruments or distance bailouts are needed.

it you want to comment something more I'm just looking at surgeries the idea of recognizing

the phase that you say it's very important to know where you are in the surgery so for the

machine is very important and now we're just watching the year progressively

leading the criteria of the critical view of safety and just

In fact, the safety has three elements.

This is very important.

Here we come in on the three components,

which is the hepatocystic triangle is very clear from all the tissue.

Second is that we have only two structures entering the gallbladder.

And the majority of people are just focusing on that.

so i think it's a good moment to comment here you can see that the surgical field has a little

it's a little bit stained by a very little some blood and still the segmentation is

good on on identifying the gallbladder and you can see that it does some mistakes on on

identifying the cystic plate in this case again this is a work in progress and and we are developing

it but on the other side every now and then you can see and i think it's quite interesting that

the machine starts to see where the cyst duct is which is the dark green segmentation you can see

where the cystic duct is supposed to be and maybe i mean you can also show the localization

is now using an aspirator device and a grasper and the in the artificial intelligence model

another one in this case is able to identify these tools and also localize them in space

space. So this might seem not of direct use. However, if we can infer automatically and

quantitatively where tools are and how tools move, we could think about building metrics

about surgical performance. So to basically correlate intraoperative performance during

during a cholecystectomy with post-operative outcomes or with the learning curves of trainees, for instance.

Both our applications, Pietro, are also no-go zones,

which would be very interesting to introduce, in fact, in this type of demonstration.

This is extremely true and it's something that we look forward to do.

So, Nicolas is referring to a model called GoNoGoNet, published on the Annals of Surgery last year together with ours, that basically was produced by an American-Canadian team of surgeon-scientists led by Amin Madani.

And basically, these models suggest where it's good to dissect and where it's not good to dissect, where it's safe versus where it's not safe.

And if you couple that with tools, you could also put another metric on how many times you go dissecting in a good spot.

You want to comment?

I'm just thinking of the surgery.

So, you can see that in fact the machine will try to find the criteria of the CVS, and sometimes

it's floating from one to the other structure, but this is not the point, I mean the machine

is looking for this element, and at some point, at the final phase of the dissection, DG-Mutair

will do a five-second timeout, so you will stop the procedure,

and that will allow the machine to really catch all the true elements

of the critical view of safety and validate the critical view of safety.

So don't look at this color changing from one structure to the other one.

The machine is just trying to detect everything.

At some point, the machine will assess together all the trick layers,

and you will see on the screen the validation of the trichet area.

So here you can see that sometimes the machine detects this artery,

you see in red.

Sometimes it's a little bit more complex.

So this is related to the phase of the dissection.

So we don't ask the machine to recognize at first the cystic artery

and the cystic tear because we have to dissect to allow the machine to do so.

but I don't know

technically speaking maybe it would be

more interesting to

load the phase

when you are approaching

the final phase

of the critical reality

but keep in mind that

the surgeon doesn't have this picture

so it doesn't interfere with surgery

so it's just on the screen

that he can't see

so it's interesting to see

how the machine is working

in effect and uh that's we are backstage exactly so we really need to stress this point that as

professor aleman was saying this is not been meant to be streamed in the operating room here

the point of this demonstration is to show basically what the machine is assessing we're not

envisioning to use this type of information to assist surgeons the segmentation i'm speaking

out. And another thing I just would like to comment, there are no easy cholecystectomy.

That's reality from a long experience. Cholecystectomy is never easy, it's very important.

The machine is a nut for the safety of cholecystectomy, but keep in mind that you still

have to do surgery dissection proper in this section and the machine will not do the dissection

for you so this is very important so it's very important because you wanted to add something

not especially you know effectively so it's only assessing the scene i think when we get a good

view of the segmentation of the cystic duct cystic artery we should describe what the colors represent

it may not be of use to everyone this is very true let's wait for the for this structure to become

come up here and then we will comment

on the, basically we

But it's

working very well. I mean, you can

see the machine is really following

the dissection and trying to find

the elements. So

the machine has to group all the

factors that

would tell, okay, this is an artery or

this is a cystic disc. So it's really

interesting to see the machine functioning.

I mean, it's magic.

Okay, I think that I am

almost not far away from the view of safety, we know that we have a fibrotic cystic duct

because the patient has a previous ERCP. We see very well the common bile duct, the artery.

Cystic duct is not totally seen, but it is in the fibrotic tissue. We have the artery

that is isolated with the hook we keep in contact between artery and cystic

duct in this case due to the fibrotic reaction of the migration that was two

months ago I will try to see if I can increase the distance but I am afraid to

have some more blood in the field so here very very taste because when we

So we're looking at the machine, and the machine can see the artery, can see the cystic duct.

And now we are implementing the machine criteria.

So the machine will say, okay, I've got the arteries, I have the two elements, I have

the cystic plate, and I have the dissection of the hepatocystic triangle.

So just so that everyone understands, C1 represents the dissection of the cystic duct and cystic artery, so the first criteria of the clinical view of safety.

C2 represents the clearance of the hepatocystic triangle, so having a window between the cystic duct and artery and between the cystic artery and plate.

And the C3 represents the dissection of the gallbladder from the...

time out from the cystic plate.

With a good time out, I think I need to show it like that.

Yes, exactly. I think here we are observing a very

nice behavior of the model, that since there is some inflammation in between

the cystic duct and artery, and we cannot really get a wide

space, a wide window in between the two, you can see

that the C2 criteria is the one flickering,

where the c1 so the cystic duct and artery dissection and c3 the dissection of the cystic

plate are almost always considered achieved again this is another nice behavior you see that now

that we have a tangential a straight view so non-anterior and posterior the c3 is no longer

visible the cystic plate is no longer visible and the machine assess it as non-achieved then when

try maybe did you not touching anything and so the machine can stabilize and we will see what

the machine is going to say so i i think that is demonstrating exactly what we're seeing so c1

so the clearance of the cystic duct and artery is clearly achieved the clearance of the hepatocystic

triangle it's flickering between achieved and non-achieved because professor muter has to

to separate with the cystic duct and artery.

And as of now, because we don't have traction,

the cystic plate is not visible.

Now it's visible again, and it's success as green,

so achieved.

So I'm extremely happy about this inference.

Again, it's so good to say when I feel so well,

this is a prototype, trained on a style of...

Yes, but it's important to say that.

Beautiful CVS, Didier. Thanks a lot.

Thank you very much.

So you did your five-second long intraoperative timeout.

You have the double check by the artificial intelligence.

Now, another thing, I think it's interesting.

You saw that the machine recognizes that Professor Moutet inserted a clipper

and moved the face change from hepatocystic triangle dissection to clipping and cutting,

which is, I think, interesting because one could think about using that information

to do selective documentation of exactly this critical moment of the procedure,

as we demonstrated with the report on the anus surgery.

And now the model for CVS, of course, was trained to detect this view up to the first clip applied on the cystic duct and after it, because you should achieve it before this moment.

is the machine is showing the cystic plate systematically so this is something that is

extremely important if you want to have a change towards the uh the uh the best uh

performance during laparoscopic so you see this uh structure is always highlighted just for the

audience the yellow highlight that is supposed to be the cystic plate of course now it's

covered by the gallbladder which is green cystic artery which is red and the

cystic duct now is no longer visible but was dark green so I think it's very nice

demonstration thank you very much for for this person i just want to congratulate the team of

nicola and people i think it's uh it's an outstanding demonstration uh there's still

a lot of work to do of course but uh this is very promising and uh i'm pretty sure that uh

chronographically we have some real support for for for the education which is really the aim

and, of course, for the safety for the patient.

This is very important.

This is beautiful.

Thank you very much.

Maybe since I think the clue of the procedure,

so the clipping and cutting of cystic duct and artery, it's over.

Maybe we could comment a little bit about what we're seeing

in terms of what it took to get there.

So, Armin, can you please toggle off the segmentation?

so this is probably a good comment

here

again this model was not trained

after the clipping and planting

of cystic duct and artery

because there is no longer needed

to assess the critical view of safety

so you saw that

the solutions were a little bit more noisy

so this model was

developed

was trained by

400

segmented manually

manually by myself and Deepak Kalapat and Armin Vardazayan.

So that's a limited number of data, 400 images.

Just to give you some ideas of where this is going.

Our most recent dataset contains 2000 images segmented manually,

which is five times more, and this dataset is not selected.

So I'm very optimistic that this AI prediction, with that limited amount of data, that the next iterations of these prototypes is going to be even more stable and accurate.

I don't think that, I don't know what would be more accurate, because it wasn't.

I'm already

Didier

merci beaucoup

thank you very much

thank you for waiting

for us

you must thank all the team

nurses, anesthesiologists

everybody makes

his effort to participate today

thank you

this is a very important

moment in the history of

cholecystectomy

so thanks to all you team

in the Strasbourg.

Very proud to be part of it.

Thanks again

and looking forward to see you

quite soon with the feedback

of the people who are watching this

procedure. Thank you, Didier.

This history

started 10 years ago.

In 2012,

we started

to recognize instruments

in surgery. It's very funny to be there

today. Thank you for the team.

Fantastic, fantastic journey and it will continue.

Thanks a lot.

And thanks also to the tech team who made the life possible today.

Thanks a lot.

Particularly to Jean-Paul Marseillais, the really ex machina of this connection.

Thank you very much.

Okay.

So Didier, we will leave you again.

Thanks so much to all your team in the hospital.

and then I think that it will want to cry because he's so pleased very very

haven't that oh you want to do a little talk about process I'm happy to give a

little talk about okay so you can give a little talk it is okay thank you very

Thank you very much, bye-bye.

Thank you all the team.

That's it?

So you toggle your slides, Petro?

I will.

People don't realize it, but

the first time we recognized an instrument automatically

was nine years ago.

And when I started,

the first time I presented the project with Padua,

we were told,

we have no interest, it will never be worth anything.

Great, thank you Jean-Paul.

By the way, Pietro has disappeared.

second to introduce myself, I'm Pietro Mascagni, Doctor from Gemelli University of Rome Cattolica

and I've spent the last three years working in the team of Nicola Padua, the CAMMA team

housed at the ESU Strasbourg, University of Strasbourg, and I've been also supervised

by a Professor Bernardo Alemagne, who is a Vice Director of the IRCAD, and by Professor

seguito costa magna in rome catholica gemelli so after this presentation i'll start by a few notes

about surgical ai so here the vision of is that operating rooms are increasingly complex

and this can be configured as a social technological process that is full of

information, which is highly effective when it goes well, but any disturbance in the flow

of this information can create a problem, an adverse event.

Just to give you some figure, a large majority of medical errors are considered to happen

in operating rooms, and 50% of them seem to be preventable.

Techniques like the ones we have seen today are pretty good at modeling information and

make sure that the right information is given to the right person at the right moment like for

instance the assessment of the cvs before clipping and cutting the cystic duct and artery that we

just demonstrated basically this enables the concept of a surgical control tower so a place

that gets information from operating rooms models it in order to monitor activity anticipate

anticipate potentially dangerous deviation from the normal course of events, intervene to prevent

adverse events, and overall optimizing the whole surgical care from training to patient outcomes

eventually. And that's what we're really looking forward. Behind all of this vision, there are

there are neural networks artificial neural networks these are computational systems

that were designed to resemble biological neural networks so our brain and that basically are able

to analyze unstructured data like images with very good performances here you can see a very

simple and didactic representation of a deep learning network so a series of layers of neurons

that detects what tools is present in the image as you saw in the demonstration basically the

cornerstone of these artificial neural networks is their ability to learn so learning is really

really the basic of machine learning and I would say a really big part of artificial

intelligence. This learning usually happens through experience, and experience when we

speak about informatics, computation, it's data. So you give a machine a data point,

an image in this case, a surgical image to be precise, then we pass through a neural

network which is a function and the machine will output a prediction in this case a wrong one

and it knows that it's a wrong one by comparing to a ground truth this is of course an

oversimplification of it all but i hope that it will deliver a message basically when the prediction

is non-correct there is an error the function updates itself so changes the connection so that

the next prediction is eventually more precise up till until this is correct like in this case

so this represents in a very simple manner what the learning process it's about if you want to

know more about artificial intelligence and surgery i invite you to to get this very nice

book produced by our friend daniel ashimoto and a very large community of surgeons and computer

scientists that distill this information for surgeons so no mathematical background

and i also invite you to to connect to edu4sds.org which is where we will soon announce a

surgical data science summer school that we hope to host next july in strasbourg and this is really

focused to teaching computer scientists and surgeons about their reciprocal needs and

solutions in order to foster innovation in surgical ai and surgical data science in general

so now i will focus with a few simple slides on what we have seen today during the live so first

First of all, we've seen an analysis of a laparoscopic cholecystectomy.

The reason being multiple.

First of all, and it's probably the more trivial, is that this is an extremely common procedure.

So we have a lot of cases of laparoscopic cholecystectomy.

Imagine in the US, they say they do a million procedures per year.

And in China, they say three million.

So a lot of opportunities to basically get data about this procedure.

And also, it's in the hands of most surgeons, which is important if we want to innovate for the vast majority.

A very connected and important point is that this procedure is in 98% of the cases performed with a minimally invasive approach, laparoscopic.

topic. And here in the connection today, we had the founders, the pioneers of laparoscopy,

but that's very important because we have digital data about this procedure. And we also have

clinically well-defined needs, like for instance, the need to prevent the visual perception illusion

causing major bile duct injury. And I would say another critical factor to have this kind of

of artificial intelligence prediction on cholecystectomy

is thanks to the team of Nicola Padua,

Professor Nicola Padua from the University of Strasbourg,

that a few years ago in 2016 published the first dataset,

a collection of 80 laparoscopic cholecystectomy videos.

And that really enabled the field

and the research of the computer scientists

that started to use this public dataset

asset to benchmark their innovation, their new models,

their new AIs.

So now let's go see what we saw today.

So probably, as I commented before,

the number one thing a machine needs to do

is to understand where a procedure is,

so the status and the evolution of a procedure.

And this is not dissimilar to how surgical trainees get acquainted with new surgical procedure.

So I'm a trainee myself, and when I need to start to scrubbing in a procedure, I didn't know.

First thing I need to learn is the steps.

So the sequence of events I need to accomplish in order to have a successful procedure.

So here I think we can see a very nice parallelism.

Machines also need to understand this in order to know where they are,

in order to then be able to produce analytics of higher surgical value.

Then we need to learn basically how to manipulate the anatomy,

how to move our instrument.

So again, the machine as well needs to understand where these instruments are in space

and what these instruments are in their function.

So whether it's a hook to dissect,

whether it's a grasper to grasp and retract, and so on.

So here again, and I like to stress it,

what you saw today are some stable AI,

artificial intelligence models that were developed throughout the years,

but unfortunately not the most recent one

because this still needs to be optimized to run in real time in operating rooms.

But what I wanted to introduce is that this kind of analytics now is moving from understanding what tools is being used

to understanding where and how it's being used up to understanding very fine-grained, detailed surgical actions.

For instance, in the latest work by Cinedo Nuoia that you can see cited here, and that's very exciting.

Machine starts to understand the single action performed by surgeons.

Then, of course, we need to know our anatomy, and that's probably one of the most important things.

And we've been training machines to understand hepatocystic anatomy, so to have a better assessment of the critical view of safety, for instance.

And this is the kind of model that you've seen during life before.

This model assesses segments, technically speaking, highlights, as we would say probably in other sectors, the anatomy.

In this case, the cystic duct, the cystic artery, the gallbladder, and the cystic plate.

Here, I would like to draw your attention to the fact that the system is not only segmenting tools and anatomy,

but also the surgical dissection, the windows in between the cystic duct and artery,

and between the artery and the cystic plate.

so this is a slightly less less it's a service it's an instance of us higher

surgical semantic because here we're speaking about this section and I'm

really excited to see where this will go in the near future when we will start to

assess a little bit more abstract instances finally as surgery we want to

to have a surgery of a certain quality

that guarantees good patient outcomes.

And also machines are starting to understand

and assess the quality of our dissection

as we have demonstrated with the critical biosafety.

So I wanna re-show this video because as you can see here,

the machine first assesses the first two criteria

of the critical biosafety as achieved,

but does not find a good dissection of the cystic plate.

So the surgeons get some feedback with the automatic segmentation at anatomy and the sex until the third criteria becomes achieved.

Finally, today, what you have seen is the optimization of all these various models on a demonstrator.

Again, this is a prototype, not for clinical use, but basically this shows, this is a step forward, the translation of these AI algorithms, because we need to be able to use them outside of lab, eventually in operating rooms, which is our final goal.

and to do that we have we have we have we need to have some good computation so we need to have

servers computers that are able to run this very computationally expensive analysis in real time

without lag and constraint space and of multiple models at the same times and this is why we're

partnered with one of the leader of these chips optimized for for deep learning inference in video

and basically here you can see from the experimental operating room a demonstration on

how this analysis can happen in real time which was also demonstrated in the live where we were

when speaking and commenting with a surgeon.

And this is, we believe, an important step forward

to the clinical translation.

And with this, I conclude the disclaimer

is no longer necessary because this presentation

was supposed to be before the actual presentation.

And unfortunately, I cannot take question,

but I would be very happy to.

I stop my slides and I eventually comment with Professor Nicolas Padua and Professor Bernard D'Alemagne to conclude this live streaming.

I think that I don't have to comment because I'm not an engineer.

Me neither, by the way.

I will let the final words to Nicolas Padua, who's really the head of all this project, and just to congratulate him for this work.

Nicolas, you have the floor.

Thank you, Bernard.

So, last few words to conclude.

So, first of all, I mean, it was a bit daring to do this demonstration today.

And I think it worked really well because when we develop AI system, you always evaluate them offline on, you know, data sets on our computers.

It's a different type of evaluation to run them live without knowing what is going to happen.

And this was the first time we did this.

And I'm very, very happy about the results of the models.

We had a lot of views where all the AI models were correct for recognizing phases, recognizing

the tool, recognizing the anatomy, and also the grading of the critical view of safety.

So I think it's really a success for this first live demonstration.

And I would like to conclude by thanking a lot Pietro, who really devoted a lot of efforts

to make the development of these AI models possible

with a clear clinical applications

and going to the last step,

which is demonstrating it on patients.

Of course, there's much more to do.

User interfaces showing that it can really impact

clinical outcome and that it has real benefits for patients,

but that's a first milestone today.

And congrats again, everyone,

and also the whole team, engineers and researchers

who really helped carry this forward.

Of course, all the clinical partners

from IHU Strasbourg, IRCAD,

and the others from Rome as well,

who helped a lot.

Thank you.

Thank you very much, Nicolas.

Thank you and congratulations for the new move.

And looking for the next stage of the development,

clinical studies that are coming for WCV.

So thank you very much.

Thank you very much, thank you Pietro, you can conclude.

Well I just conclude saying that I'm extremely happy that in very few years I would say because

I mean pioneers like Nicola Padua started this research with other very well-known groups

some years ago where this application was not really perceivable or not many believed in it

And I think it takes a vision to be able to pursue and run this marathon that eventually leads to something meaningful for patients, surgeons and healthcare system.

and the whole surgical data science community it's a very nice one that is

really taking this concept to art and is trying to move and demonstrate with

thorough assessment clinical metrics vision a concept that is very often too

much hyped in my opinion but that has realistic potential to be meaningful to

people thank you but i i think that the most important uh is also to thank very much uh

professor palazzini from roma uh we know him for a while and uh i have to say that when

tietro came with the id uh he was extremely enthusiastic and uh put all his efforts to

to allow this session to be transmitted to the famous video sessions in Roma.

So thank you, Professor Palazzini.

Thank you very much.

I hope to see you quite soon.

Bye bye.

And thank you everyone who is watching from the World Congress of Endoscopic Surgery hosted

by EAS in Barcelona this year.

And I would also...

We have to work now.

We have to work now.

but I want to thank EAS as well because they first co-founded partially but importantly

this project back in 2018 with a research grant and that really was one of the enablers of

this innovation at least in terms of motivation from the team side. Okay so we're going to the

congress because we have to we have to do the congress now thanks everyone thanks bye