contador web Saltar al contenido

Free 3D Computer Graphics Applied to Health Sciences

I have always enjoyed learning new things and also sharing the knowledge gained. Since 1996 when I took my first course involving computer science (microcomputer operator) I knew that computer graphics would revolutionize with its many possibilities.

In the late 1990s there were not many didactic possibilities. The internet wasn't as wide as it is now, especially in the city where I live, so sites like Youtube and others that help you understand any topic were out of the question. It was necessary to learn from the manuals, these written in English, a language I did not understand and of course, the other option was by trial and error.

In this scenario, the programs that stood out were the most accessible and easy. I remember meeting one of Floorplan Plus 3D, which fit on a floppy disk and allowed us to model a house in 2D and at the same time it was converted into a three-dimensional scene. It was fantastic, but people wanted more, not just a design-like 3D, the clientele wanted shadows, sparkles and reflections.

Given this I extended the studies and arrived at Corel Dream 3D. With it I was able to create scenes with shadow and texture, which caused great amazement at the time and gave me the first gains with 3D computer graphics.

After that I studied 3D Studio Max and other tools, but as time went on, it became clearer that I couldn't participate in all this, at least the way I wanted to.

In the early 2000s I learned about Linux and was amazed. My first experience was not the best, since I could barely move the system, but I did not give up and in 2005 I started to use only it on my computers.

I am very grateful the architecture area, after all it was my gain for many years. However I wanted more challenges, I moved to the advertising area and finally in 2011, after a traumatic episode where I reacted to a robbery and took a scratch shot in the head, I decided to dedicate myself to the forensic facial reconstruction camp, this opened me up many doors in archeological projects and paved the road towards the health sciences.

In 2014 I met Dr. Everton da Rosa, a buccomaxillary surgeon who came to me to learn how to work with Blender and use it in planning orthognathic surgeries, procedures that correct facial deformities in adults.

This partnership was evolving and as we posted our progress, many colleagues from Dr. Everton began asking for courses. We did so and created the first Practical Course on Computer Graphics Applied to Health Sciences.

Everything went very well with the course, but we realized that the health care people were suffering a little to absorb the content. Lacking capacity, on the contrary, the problem was much more linked to lack of time. Everyone who has studied computer graphics knows that dedication and coexistence with the programs are necessary for their domain.

What we see, at least in Brazil, is a framework where health specialists need to work in various jobs and also take care of their consultations. There is little time left to devote to a new tool and by the way, the technology market changes very quickly and it is impossible to keep up with all the possibilities it presents.

To solve this question I started to create a series of pre-configured files for the course. This I've been doing since 2001, when I started teaching computer courses. But just providing these files didn't solve the problem of knowledge absorption.

I will explain better. Imagine that a specialist wants to do facial surgery planning. He will first need to convert a CT scan to a 3D surface corresponding to the desired anatomy, for example, skin and bones.

Most digital CT scans are DICOM files, a sequence of grayscale images that contain slices of certain areas of the body. In this case, the specialist receives the tomography (tomography) of his patient's head. This is nothing more than a three-dimensional matrix composed of a series of stacked images known as voxel data. Roughly, the hard parts such as bones are light, the hollow or less hard parts tend to be darker. Then, by choosing an area of ??interest, taking into account this intensity, it is possible to filter out a specific part of the anatomy.

The first thing the expert needs to do is open this tome in 3D visualization and reconstruction software. He then selects the area of ??interest, segments this area, and finally generates his 3D. Then he needs to export this file and import it into Blender to proceed with the osteotomies, which are nothing more than bone cuts.

So far so good, difficult but not impossible. The problem we are talking about organic 3D meshes from tomographic reconstruction are very heavy and the cutting calculations that are made by boolean do not work with Blender's native tools, which are designed for less complex graphics.

How to solve it? Simple, looking for an external alternative. In this case through the standalone boolean calculus program called Cork. It is a command line-accessed program that proceeds with complex booleans and really does the job. The problem lies precisely in the command line part, teaching this to health professionals who had never worked with exhausting 3D for both them and the teacher.

Are you, dear reader, understanding the size of the problem? For we not only understand it, but we live it in the skin.

An activity that was to be quick, such as importing a volume and starting to work with it, became a never-ending activity, and this was not even the beginning of the total procedure, which still involved moving these parts and recording it. It is dynamic for studying the surgical approach, as well as creating guides that would help practitioners know exactly where to cut bones.

The first addon we never forget

I always liked programming. I had never been seriously involved with it, but I kept reading about this technology, familiarizing myself with the concepts covered.

One day, after our first 3D health science course, I decided to try to stack the most commonly used commands in one corner of the Blender interface. After years and years of teaching courses I knew very well the students' difficulties and the goal was to put in a space almost everything they needed.

I watched a series of video tutorials, read a lot about it and set up this little interface by leaps and bounds. The days went by and I realized that I could extend the capabilities of our humble addon. More than simply stacking buttons, I began to group functions into a single button.

For example, most health professionals want to print the results of their work to study them better. A lot of people think that just taking any 3D volume and sending it to a printer is materializing it directly, but things don't work that way. The file needs to undergo a treatment that basically clears any incongruities from the surface and closes all holes so it can go on to print.

In Blender you can clear a mesh via a modifier called Remesh. With it the surface is converted into a series of 4-sided planes, the parts that are separated can be erased and the object is ready to be printed.

What I did in the addon was create a button that assigns the selected object the Remesh command already with all the necessary settings to allow a good impression.

Now let's imagine the solution to the problem proposed above when we discuss the difficulties of caring for a tome and proceeding with osteotomies. Besides being able to group a series of commands in a button, we can also create sequences of commands that even call external programs.

This is because Python, which is the Blender scripting language, has a number of libraries for all sorts of purposes. One of them is the possibility of running external applications with the necessary arguments, as if it were directly from the command line, that is, we can create a button that does things that Blender does not like importing DICOM files directly in 3D and also proceeding with cuts. complexes using a robust boolean algorithm!

This is exactly what we did in the OrtogOnBlender addon.

The OrtogOnBlender

OrtogOnBlender works as a composite list with the sequence of steps to be followed by the expert. It is constantly evolving and because of this we have decided to focus the documentation on editable and centralized files so that the user will always have up to date information about the addon. For installing on Windows, there is a step by step that can also be followed by Linux users: https://goo.gl/hZvakD

Below is a brief description of the addon sections.

Never mind

As discussed above, before we can begin surgical planning we need to reconstruct the parts of interest from a computed tomography.

Some time ago this task was made up of many stages, which proved to be quite tiring for those who were beginning their studies. After much research and testing, we found Dicom2Mesh, an application that reconstructs an STL (3D file format) from a sequence of DICOM files. Sounds like magic! But of course, pure technology. With a command line you can tell where the DICOM file directory is, what area of ??interest you want, and what the output file will be.

Never mind

Tomography Reconstruction

What we did was create a sequence of commands that generate 3D from Dicom2Mesh, matter to Blender and still relate skin to bones, so when the user moves the skull the skin follows it, even though it is an independent object. .

Graphic References

It is a set of lines that, while seeming simple to a contiguous user of 3D modeling programs, complicated to set up by a beginner, while making it indispensable for aligning the skull to a known plan.

Import 3D Tome / Molds

There are cases when the addon user either prefers to rebuild the mesh in an external program like Slicer 3D or the like, or already has this rebuilt mesh. It can then import the file that is almost always an STL.

More than that, this section also serves to import moldings from the upper and lower arches. In orthognathic planning this is often necessary since teeth reconstructed by tomography may have distortions caused by dental restorations or appliances.

Zoom Scene

One of the first things a user of 3D modeling programs learns from zoom tools is scene visualization.

However, operating systems do not always allow the use of mouse or shortcut keys as they should. Added to this is the fact that some keyboards do not have the side numeric keypad, which is essential for zoom and display commands.

The Blender interface allows us to move sections up and down, allowing the user to keep visualization tools always close to the current working step.

Create Photogrammetry

This is another highlight of OrtogOnBlender. Just set the directory where the photographs of the face are located and click on the desired algorithm that photogrammetry scanning happens automatically.

The addon then imports the resulting file and even centers the zoom on the scanned part.

Start Photogrammetry

By clicking this button the photogrammetry calculations are made by a joint solution between OpenMVG and OpenMVS.

SMVS + Meshlab

By clicking this button the photogrammetry calculations are made by a joint solution between MVE / SMVS and Meshlab.

Why offer these two options and not just one?

Why doesn't a tool always work in every situation. Offering two options gives the user a greater chance of success. For example, to scan faces SMVS offers better results most of the time. When we talk about digitizing objects like templates and skulls, OpenMVG + OpenMVS has been doing better.

To further illustrate, there are two experiments involving the tools provided by the addon compared to closed applications.

General Protocol for Face Scanning for Orthognathic Surgery and Rhinoplasty Planning – Tool Comparison

Could open photogrammetry be an alternative to 3D orthodontics?

Align Faces

Free photogrammetry tools provide excellent results, however they are deficient in alignment and scaling. Just thinking how to solve the problem that this section was created.

Alignment and Resize divided into three steps. The user enters edit mode and selects three face points. The first two are of a known measure, such as the distance between the eye limits, for example. Aligns with the camera: By clicking on the button the triangle created by the three points aligned with the camera, then we know that this object is the alignment parameter. Actual Measure: Here the user puts the actual measure of the distance in mm. Aligns and resizes: Knowing the alignment and scaling parameters, the object is finally aligned relative to the global origin (0,0,0) and scaled to the actual scale.

Dot Alignment Here the user aligns the photogrammetry-scanned face with the reconstructed face of the computed tomography.

Why align a structure that already exists? Simple, the structure from tomography contains no texture. Already the digitized photogrammetry model contains the texture of the patient's face. Even if it is the same face, the absence of texture causes strangeness as information about the patient's facial structure is lacking. how to compare a real face with a monochrome statue. In other programs the user may project a photo on CT, but this implies potential problems, the main one being that a front photo does not contain lateral information and some regions may run out causing strangeness as well. What's more, the photogrammetry is so simple and the texture is so good, it makes projecting images unnecessary.

Iimport Photogrammetry

The addon offers the user the option to import a photogrammetry, or even scan from external software.

In addition, after the scan is generated or imported, it is necessary to select the region of interest. The easiest way is to create a side circle that will serve as a parameter for a cut.

The addon not only cuts the facial mesh, it automatically erases the excess and the created circle.

Import Cephalometry

In the planning of orthognathic surgery there is no consensus on the best method of head alignment. In view of this, we have made it available to the user to import a digital cephalometry image so that it can be used as an alignment parameter.

Osteotomy

This section contains the osteotomy tools or bone cuts. Through these cuts, specialists will be able to reconfigure the patient's face to solve respiratory and structural problems.

The first part groups together a series of buttons that create predefined section plans. The user can also proceed with boolean calculations and even separate osteotomies automatically.

The second part consists of the osteotomy configuration tools. The user not only names and pigments each of them, but already harnesses the dynamics of the soft.

This is one of the smallest sections, but interestingly it contains the largest code snippet.

This explains why the Soft Dynamic Setup button groups a complex sequence of commands that create areas of influence and deformation on the face by reference to the volume of osteotomies.

It is fascinating to attest that years of study could be summarized in just one click.

Splint Creation

The reference the surgeon has to making cuts and fixing osteotomies in the real world is called splints. They work as surgical guides having as a parameter the snapping point of the teeth.

OrtogOnBlender allows the specialist to control the timing and displacement of osteotomies. By working on these concepts the expert can create a splint based on both jaw and jaw motion.

The Challenges in Implementing OrtogOnBlender

Despite all the work devoted to the addon it presents a number of challenges and points that need to be improved.

Installation

– Problem: Even with all the facilities present, it is a hard job for a novice user to install OrtogOnBlender. Full deployments are available for Linux and Windows already compiled, but not yet ported to MacOSX forcing users of this system to compile a number of applications through Homebrew.

– Solution: Those interested in using OrtogOnBlender can download or purchase Linux 3DCS, a distribution installed directly on a bootable USB stick.

Soft dynamics tied to bones

– Problem: The movement of osteotomies is done through bones. This can be a challenge for novice users.

– Solution: We are already implementing a system based on the direct movement of osteotomies. See the previous operation here: https://youtu.be/rFCZL0xeOI4

Automatic alignment of osteotomies and capture of known points

– Challenge: Some orthognathic surgery planning applications offer the possibility to automatically align the osteotomies, as well as capture a series of points (placed by the users), informing how these objects were displaced.

– Solution: Although Blender offers the movement and rotation data of an object, we are implementing a system that captures this data at specific points. This implementation will also work to align or pre-align some parts if necessary. See the previous operation at the following link: https://youtu.be/an5XXhqu8Xw

Linux 3DCS, a simple and robust solution

As discussed above, one of the major issues related to OrtogOnBlender installation lies in the need to configure other addons and install or even compile a number of applications.

Installing what is available is an acceptable task even for a novice user, but asking this user to compile a program is a little too much. So how can we solve the lives of those who would just like to test the addon without having to waste an entire afternoon setting it up? Very simple.

I have been using Blender since 2005, as I commented here. One of the great advantages of this system is its flexibility. This means that we can adapt it to many unusual situations. It is possible, for example, to install Linux on a USB stick as if it were a hard drive and this opens the door to many possibilities.

The Linux 3DCS project was born from the need for our students to have in their hands a complete and available system, based on free software, without having to download a series of programs.

We started the project as a test and now we are giving new students USB sticks with the system installed. Just set up the BIOS quickly and they can start enjoying a fully configured workflow, with all the programs needed and happy that they don't have to invest their precious time in setting all that up.

But what about users who are not our students and who want to use Linux 3DCS? Well, they can:

Download an image available for download and clone on a USB stick;

Download the image and run on a virtual machine;

Ask our team to record a USB stick and pay for the service;

Take one of our classroom courses 🙂

For those interested in this project, the link for more information and download is here: https://github.com/cogitas3d/Linux3DCS

Conclusion

We are still crawling in this wave of Linux software development and distro, our alternatives are not those standardized by the market or less by the community, but above all, we are delivering two things:

1) Projects that are open and available for download;

2) Documentation required for installation and understanding of the tools involved.

We hope to maintain that animation and focus needed to solve problems one by one and be able to provide tools that are useful to society and really make a difference.

Grateful for the reading.

Cicero Moraes 3D Designer

_____________________________________________________________________________

See an error or would you like to add any suggestions to this article? Collaborate, click here.

Rate this post