What are the tricky parts in developing an application for designing braces based on 3D jaw models?

We are developing an application that will help to correctly model braces according to the current position of the patients' teeth for one of our clients in the field of orthodontics. What were some of the challenges we had to solve in the development of this application for iPadOS, MacOS and Windows?

It is not easy to develop an application for orthodontists which uses its own algorithm for marking individual teeth on a 3D model of the human jaw. We wanted the application to be accurate and, most importantly, practical in increasing the speed of the orthodontist's work.

What is at the core of this application?

For the braces to function properly, it is necessary to ensure the fluid movement of all the teeth from their original state until they are completely aligned. The application creates a timeline of tooth movement based on their size at each individual step decided by the orthodontist. They should set the tooth movement schedule quick enough so that the treatment does not last too long and the patient saves money, but at the same time gentle enough so that the jaw does not suffer from the pain of excessive changes.

The biggest challenge - separate the teeth models

Each model of a jaw consists of a single model consisting of the teeth and gums together. The foundational step that needs to be solved is to separate the tooth models, so that the dental specialist can work with them and design their optimal movements to treat their patient.

A trivial solution to the problem would be to let the user mark all the teeth point by point. It is quite easy to implement, but very inconvenient to use. As everyone knows, an adult has 32 teeth, and due to the relatively high detail of the model from a 3D scanner, this process would be very time consuming. Therefore, creating an algorithm that finds and marks the teeth themselves seems more user-friendly. The user would only need to help by entering a few points.

What procedures did we try?

1. The marking of distal and mesial points

The first method we tried was to mark the points of the teeth on the extremities, the so-called distal and mesial points. These points serve as opposite ends of a circle according to which the rest of the teeth are marked. This seemingly small improvement has greatly simplified the marking phase. However even with this improvement, the app was still a long way from being really comfortable to use. Although the circumference of the jaw is usually round, it is not a perfect circle and so most of the points still needed to be adjusted manually.

A closer look at a jaw scan always shows a sharp line at the border between the tooth and the gums. This can be used to position points more accurately. Firstly, however, these border lines need to be found and transferred into a structure that the program will be able to work with.

• ### The search for convex and concave points

We evaluated each point of the 3D model according to an algorithm that calculates the average distance of all adjacent points to the surface at the examined point and with the same normal vector as the normal vector of that point. This distance had the same orientation (meaning, it was positive when the point was above the surface and negative when it was below the surface). So, if the point had a positive rating, it was part of the convex section of the 3D model, and if it had a negative rating, it was part of the concave section. The higher the rating, the sharper the change in area.

• ### Combination with points from a circle

We could now combine the selection of points from the circle with the alignment of the points with the highest convex rating. The algorithm began tracing points from a defined distal and mesial point. From these points, it selected a set of neighboring points, from which it determined the one with the best rating and which had the shortest distance from the leading circle possible. The algorithm found a continuous point between these points and the whole cycle was repeated until the paths found were connected from the distal and mesial points. Thus closing the circuit.

Functional, but…

This method enabled a sufficiently fast selection of teeth for the user, it was not computationally demanding and any incorrect markings could be easily corrected. However, the algorithm was highly dependent on the correct placement of the starting points of the teeth. If one of them was already outside the convex boundary, or the boundary deviated significantly from the leading circle, the algorithm was not able to find the path of the tooth boundary and only the selection of points along the leading circle was used. Fortunately, this was not the case very often, and the algorithm was therefore efficient enough to select teeth.

Although initially the algorithm found the teeth relatively successfully, with increasing available jaw scans, its error rate gradually increased. Even the adjustment and tuning of the selection preferences according to the convex area or according to the leading curve could not increase the success dramatically, so it was necessary to devise another way to detect teeth in a 3D model.

2. Selection using an area

When we concluded that the path boundary selection on the 3D model did not achieve the desired success, we decided to test the area selection. In this case, the user selects only one point, the middle of the tooth. The algorithm then begins to expand and selects points in the set. If it selects it, the algorithm will determine its size according to its evaluation from the previous procedure. As a result, the expansion always stops around the tooth-gum line. There is no need for a leading circle, because the selection always takes place inside the tooth. However, areas with a convex rating do not only mark the boundary between the tooth and the gums, they also include the grooves and irregularities on the tooth itself.

• ### How to prevent rapid expansion

In the first steps, the expansion may stop too quickly if the starting point is inserted at a point where the tooth is very uneven. To avoid this, there is an area around the starting point where the rating check is skipped and each point is automatically selected. The user sees the size of this area by showing a semi-transparent sphere and has the option to adjust its size using the slider. Due to the fact that people usually have teeth of approximately the same size, it was possible to predefine the sizes of these areas according to the type of tooth currently selected.

• ### Conversely, how to stop expansion in time

When carrying out the last steps, it may happen that the border area of the tooth is not very pronounced at some points and will not be evaluated sufficiently, in order to stop the expansion, and thus the selection of the tooth. For this case, the user has 3 predefined values that will help them determine the threshold of when the point should be included in the selected expansion and when it shouldn’t. If the selected threshold still isn’t enough, the expansion is limited to a fixed distance of 1 cm from the beginning point. During each selection, the user sees a coloured preview of which area matches and is selected and which does not, and how adjusting the two values will work.

3. Smoothing out the selection

The final step is to smooth out the selection. As already mentioned, the evaluation area is not perfect. Thus, many parts of the selection can be created without selected points (holes in the selection) and the border area can be created several points different from the actual tooth and gum border. To remove holes, we select a path from the currently found boundary and adjust it so that its 2D projection represents a convex polygon. Finally, the points of this path are aligned to the nearest point with the maximum rating.

After applying all these additional steps, the resulting path traces the line between the tooth and the gums very faithfully. Thanks to the easy and understandable setting of two values by the user, the success of this new method is much higher than it was in the first version using just the circle. At the same time, suitable default values lead to a successful result without the need for manual adjustment, so using this method is faster for the users.

In conclusion… it was worth it

The problem of finding the edge of the teeth and gums is one of the many problems we solved with our application. However, it cannot be said that if we started directly on the second path, which in the end proved to be more suitable, the development would be more efficient. Many solutions are built on top of previous experience and the same wisdom rang true in our case.