Augmented reality (AR) and virtual reality (VR) are two technologies we believed possible only in sci-fi movies until not long ago. Now they’re slowly becoming part of our day-to-day. However, despite their advances, they still fall short when reaching multilingual audiences. This is where real-time translation comes in!
We previously outlined real-time translation benefits for multiplayer games. Here, we’ll explore its integration with AR and VR in video games.
What exactly are AR and VR?
AR is everywhere now, but you might not realise it: Filters on your Teams meetings, or Facebook Messenger calls that change your background or your appearance; a QR code with “enriched” views of a product, such as furniture, food or clothes; or some books that allow you to see the characters in 3D with the help of your mobile phone or tablet. So what is AR exactly? It’s an enhanced view of reality through images, video, or audio through different devices.
Similar to AR, VR shows an alternative reality; however, this reality is not based on the real world. It might be entirely independent. It also requires more specialised equipment such as VR headsets or VR connected to a PC or VR consoles. If you’ve had the opportunity to try this already, you might have been able to tour a World Heritage site or play a VR game such as Beat Saber, Resident Evil 4 or The Elder Scrolls V.
The Rise of the AR/VR Market
Technological advances have enabled AR/VR companies to expand implementation in gaming, healthcare, and education, among other sectors. AR/VR is expected to show a CAGR (Compound Annual Growth Rate) of 22% between 2025-2030, with $200.87 billion in revenue by 2030.
AI integration, for instance, has also improved gesture recognition in VR, allowing users to interact better with their games and applications without other controllers. This and other developments promise to increase VR use and applications in the next years.
Source: Innowise
Benefits of Real-Time Translation in AR and VR
The benefits of real-time translation in AR and VR are very similar to other areas, such as e-commerce and entertainment. However, consider the following benefits for AR/VR gaming:
- Efficiency and speed in communication: Real-time translation means skipping long localization pipelines. This is especially useful in multiplayer games or VR live events, where simultaneity is key.
- Richer experiences and player engagement: These technologies are all about immersive experiences. It is not only about imagining seeing yourself in a game scenario but actually being in it. Real-time translation can support better interaction for non-English speakers, thanks to voice recognition integration, for instance.
- Accessibility and inclusivity: The possibility of having closed captioning and subtitles instantly is a great advantage for players with disabilities. The technology that powers real-time translation can also help these players enjoy the experience without extra effort.
Source: InvoZone
Current Solutions
Some of the most popular solutions out there include:
- Google Lens: This app can be used with any smartphone. It allows users to find objects/places on the web or do instant translation of text with just the pictures on their mobile’s camera.
- Meta Orion glasses: META’s new AR glasses –not yet available to the public– give an enhanced view of the real world. It’s integrated with META AI, META’s virtual assistant.
- Microsoft HoloLens 2: This mixed-reality headset integrates gesture recognition and eye tracking. Although this headset is a huge advance in terms of mixed reality headsets, the HoloLens series is being discontinued.
- Meta Quest 3: The best-selling AR headset right now has improved resolution and processor, allowing a better integration to PC and a richer experience in games and virtual social events.
- Play Station VR2: 2nd in popularity after the Meta Quest 3. This VR headset promises a better immersive experience for players with enhanced visuals, eye tracking and movement tracking with controllers.
AR/VR Games
If you also jumped on the Pokémon Go craze, you know how AR games work already. This game gives players the experience of being a Pokémon trainer and finding Pokémons all around the players’ locations. Players just need to download the game on their mobile phones and use their cameras to see and catch the Pokémons around them. The game reached such popularity that some brands started using its capabilities for marketing purposes.
Other AR titles include Angry Birds AR: Isle of Pigs, which allows fans of this franchise to play the game in AR. Players can shoot towers and structures with their mobiles, allowing a more immersive experience when aiming and exploring the setting. Some games can be played with mixed reality headsets such as the HoloLens as well.
Unlike AR, VR requires more specialised equipment, as we outlined above. Some of the most popular titles include famous series in other consoles such as Half-Life, Minecraft, or Resident Evil.
The Elder Scrolls V: Skyrim VR Edition, for example, is a game that can be played on different VR consoles. This popular RPG (role-playing game) series allows players to enjoy the lands of Skyrim, complete tasks, and collect equipment for their character. Because of the many possibilities and settings, it presents an engaging, immersive experience for the series’ fans.
Challenges and Considerations for Localization
Despite its potential, real-time translation in AR and VR faces several challenges:
High-speed connectivity issues
In general terms, high-speed internet connectivity is crucial for a seamless AR and VR experience. Low connections disrupt videos, audios and connectivity with other players, which breaks the immersive bubble. The same applies to translation; high-speed connections such as 5G are ideal to keep low-latency translations.
Accuracy and Context
We touched on this in our previous article. Capturing nuances, idiomatic expressions, and cultural contexts remains challenging for AI/MT systems in gaming.
Ethical Considerations
Some of the ethical concerns regarding real-time translation are:
- Privacy: Gaming data and users’ data must remain private and only available to the users and not third parties.
- Data collection and usage: What data is being collected and whether it is being used for training purposes (AI agents and language), marketing, or other purposes.
- Potential for misuse or misinformation: Minimize bias in MT and language models to avoid the continuation of stereotypes and issues related to the spread of misinformation.
Besides these considerations directly related to MT/AI, there’s an additional layer of security concerns because of AR/VR technologies’ privacy.
Upcoming Developments
The future of real-time translation in AR and VR is promising:
- Multilingual Avatars: AI-powered avatars that can communicate fluently in multiple languages.
- Context-Aware Systems: Improved AI models that adapt translations based on context or user preferences.
- Integration with Wearables: Enhanced AR glasses capable of advanced real-time translation for everyday use.
Source: Yord
As AI technology continues to evolve, we can anticipate even more sophisticated solutions that will redefine communication in virtual environments, fostering cross-cultural understanding and expanding market opportunities across the globe.
