Linde Virtual
Academy

Darian and his team have come up with their own version of a Universal translator, inspired by the very one used in Star Trek. Here’s the story of how his team came up with this idea, which was later refined and released by none other than Google.

Darian is an XR expert working in this field for over 15 years. He and his team created an application that breaks language barriers. (Photo: Delta Reality)

Darian, thank you for joining us today. Tell us a little bit about how you came up with this brilliant idea.

You see, virtual communication and language barriers are something that we, as an XR studio, are very passionate about. This is partly due to a realization that the use of virtual and mixed reality could change our methods of communication. To communicate successfully, three elements are necessary:

1. to be in the same place
2. to speak the same language
3. to have the same cultural background. 

The first element has already been solved by technology. For instance, if we consider a phone call to be 50% of the benefit of a face-to-face meeting, a remote meeting to be 75%, by adding virtual or augmented reality, this figure becomes 95% for providing the same feelings that a face-to-face meeting provides.

From this, we can conclude that the whole problem is 100% solvable by technology, right?

That’s right. In fact, going back to the three elements, there are two that are still seemingly unsolved, right? Even these already have working solutions thanks to technology. When it comes to speaking the same language, this is where the Universal translator comes into play, while ensuring that we have the same cultural background can be achieved if AI is used exclusively during the conversation.

That’s an interesting explanation. Your product is therefore meant to remove all three obstacles to communication. What was the starting point of the development, and how long did it take you to complete the product?

It started with Nreal’s glasses, which are less bulky than those that used to be available, and the advancements in the tech have been significant. My idea was to use an AI service to create subtitles that were visible through Nreal’s glasses. For this, we used Microsoft’s Azure Cognitive Services, particularly their speech-to-text and speech translation system. A two-man team finished the prototype in just ten days. 

Nreal's glasses were the starting point of the whole idea. (Photo: Delta Reality)

That seems very fast indeed. So how does it work?

At the start, I assumed it would go quickly, but with the existing components we used, it really surprised me how simple it was to create something. The whole process consists of four steps: 

1. Speech is captured through cognitive services. 
2. That same speech is translated and turned into text. 
3. Subtitles visible through the AR glasses appear.
4. Face tracking helps to position these titles directly under the speaker’s face. 

This enabled me to talk to my Italian-speaking colleague, and although I don’t speak a word of Italian, I knew exactly what he was talking about. There was still some polishing to be done to the prototype, but as it was only part of an extensive application we were creating, this was on our roadmap anyway. We never released the translator to the public as a final product. Here is the preview.

We've been working on a Universal Translator; an app created with an idea of changing the way people communicate. This is just a part of the bigger app we are creating. Let us know your thoughts!@nreal#virtualreality #VR #augmentedreality #AR #mixedreality #MR #nreal pic.twitter.com/C4VkkZqEnA

— Delta Reality (@DeltaReality) August 30, 2021

Google launched their AR translator just nine months after you released your clip via social media. How did you feel about that?

Yes, in fact, that is true! As I mentioned earlier, once we started creating our version of the Universal translator, we realised how easy it was – the chances that some big players were already developing this idea were extremely high. After all, the Universal translator has long been a dream of sci-fi fans, so the question wasn’t so much if but when. Knowing that the foundation of the product is translation, we had a feeling that either Google or Microsoft would be working on such an idea, and we knew from the outset that we couldn’t compete with them. After all, the idea behind our project was not to launch it as a final product but to integrate it into the metaverse we’re working on.

One last question and we’ll leave you be. In the future, can we expect the Universal translator to be used in various projects?

For now, we only plan to use it in the metaverse, but who knows where we will end up. The thing with XR is that we are always coming up with ideas to reuse, revive and upgrade our old projects. For this one, we’re sure that it won’t see the light of day as a standalone product, but we will definitely use it again. 

Judging by some of the most recent applications we’ve seen, AR holds great potential for solving not just language barriers, but many other problems that humanity faces today. Perhaps in ten years’ time, our children won’t be using books or tablets at school but a pair of AR glasses instead. Or maybe business meetings won’t be held in person at all anymore. It is certain that solving such an important barrier as that tackled by Delta Reality’s Universal translator and the AR Translator by Google is one thing humanity needed to tick off its list. We will certainly be keeping an eye on this one.