Augmented Reality Language Learning Application
Lingua AR is an immersive AR mobile application that helps users learn new languages by detecting real-world objects and providing instant translations, example sentences, and audio pronunciation. Built with Unity and powered by Niantic Lightship ARDK.
Screen_Recording_20251214_130116_Lingua.AR.mp4
Lingua AR enables bilingual language learning where users can:
- Learn a target language (e.g., Italian) while speaking their source language (e.g., French)
- Point their camera at real-world objects to get instant translations
- Save vocabulary words for later review
- Listen to native pronunciation
- View contextual example sentences
Supported Languages: English, French, German, Italian
Platforms: iOS (ARKit), Android (ARCore), VR (OpenXR), Unity Editor (XR Simulation)
- Uses Niantic Lightship AR for advanced object recognition
- Detects 80+ common object categories (furniture, animals, vehicles, etc.)
- Confidence-based filtering (0.6 threshold)
- Visual bounding boxes with color-coded labels
- Source Language: Language you already speak
- Target Language: Language you want to learn
- Offline translation database with 4 languages
- Displays: "Source → Target" format (e.g., "la chaise → la sedia")
- Manual save: Tap objects to add to vocabulary
- Persistent storage via JSON
- Delete unwanted words
- Track detection count and last seen time
- Word count badge indicator
- Context-rich example sentences for each word
- Bilingual display (source → target)
- Database of pre-written sentences
- Text-to-speech for target language
- Native platform integration (iOS/Android)
- Instant playback on demand
┌─────────────────────────────────────────────────────────────────┐
│ LINGUA AR │
│ (Unity 6000.2.10f1) │
└─────────────────────────────────────────────────────────────────┘
│
┌─────────────────────┼─────────────────────┐
│ │ │
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
│ AR │ │ UI │ │ Data │
│ Systems │ │ Layer │ │ Layer │
└─────────┘ └─────────┘ └─────────┘
│ │ │
└─────────────────────┴─────────────────────┘
│
┌─────────┴─────────┐
│ │
┌─────▼─────┐ ┌────▼────┐
│ Niantic │ │Platform │
│ Lightship │ │ Services│
└───────────┘ └─────────┘
┌───────────────────────────────────────────────────────────────────────┐
│ AR DETECTION LAYER │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ ARObjectDetectionManager (Niantic Lightship) │ │
│ │ - Hardware camera input │ │
│ │ - ML object recognition │ │
│ │ - 80+ object categories │ │
│ └──────────────────────┬────────────────────────────────────────┘ │
│ │ │
│ ┌──────────────────────▼─────────────────────────────────────┐ │
│ │ ObjectDetectionSample.cs (Controller) │ │
│ │ - Filters by confidence (0.6 threshold) │ │
│ │ - Fetches translations (source + target) │ │
│ │ - Emits OnFoundItemAtPosition event │ │
│ │ - Can pause/resume detection │ │
│ └─────────┬──────────────────────────────────────────────────┘ │
│ │ │
└────────────┼───────────────────────────────────────────────────────────┘
│
├──────────────────────────────┐
│ │
┌────────────▼──────────┐ ┌─────────────▼──────┐
│ UI VISUALIZATION │ │ TRANSLATION │
│ │ │ │
│ DrawRect.cs │ │ TranslationDB │
│ - Object pooling │ │ - Offline lookup │
│ - Bounding boxes │ │ - 4 languages │
│ - Labels │ │ - 80+ words │
│ - Color coding │ │ │
└───────────────────────┘ └─────────────────────┘
┌───────────────────────────────────────────────────────────────────────┐
│ USER INTERACTION LAYER │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ USER TAPS DETECTED OBJECT │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ UIRectObject.cs (Clickable Rectangle) │ │
│ │ - Category, Translation, SourceTranslation │ │
│ │ - Confidence, ScreenPosition │ │
│ │ - OnPointerClick handler │ │
│ └──────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ ObjectSelectionManager.cs (Focus Handler) │ │
│ │ - Focuses clicked object │ │
│ │ - Hides other rectangles │ │
│ │ - Fires OnObjectFocused event │ │
│ │ - Creates DetectedObjectData │ │
│ └──────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ ActionMenuPanel.cs (Action Menu) │ │
│ │ │ │
│ │ ┌────────────────┐ ┌──────────────┐ ┌────────────┐ │ │
│ │ │ Pronunciation │ │ Examples │ │ Save Word │ │ │
│ │ │ Button │ │ Button │ │ Button │ │ │
│ │ └────────┬───────┘ └──────┬───────┘ └─────┬──────┘ │ │
│ │ │ │ │ │ │
│ └───────────┼──────────────────┼────────────────┼──────────┘ │
│ │ │ │ │
└──────────────┼──────────────────┼────────────────┼────────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────┐ ┌──────────────┐ ┌──────────────────┐
│ TTS Manager │ │ Examples │ │ Vocabulary │
│ (Platform) │ │ Panel │ │ Manager │
└───────────────┘ └──────────────┘ └──────────────────┘
┌───────────────────────────────────────────────────────────────────────┐
│ VOCABULARY SYSTEM │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ VocabularyManager.cs (Singleton) │ │
│ │ - SaveWord(category, translation, languages) │ │
│ │ - DeleteWord(category) │ │
│ │ - GetAllWords() → List<SavedWord> │ │
│ │ - IsWordSaved(category) → bool │ │
│ │ - Fires OnVocabularyUpdated event │ │
│ └──────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ┌───────────┴───────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌────────────────┐ │
│ │VocabularyData│ │ SavedWord │ │
│ │ │ │ - category │ │
│ │- Contains() │ │ - translation │ │
│ │- GetWord() │ │ - source/tgt │ │
│ │- AddOrUpdate()│ │ - timestamps │ │
│ └──────────────┘ │ - timesDetect │ │
│ └────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────┐ │
│ │ JSON Persistence │ │
│ │ Application.persistentDataPath │ │
│ │ /vocabulary.json │ │
│ └──────────────────────────────────────┘ │
│ │
│ Subscribers to OnVocabularyUpdated: │
│ ┌────────────────────┐ ┌──────────────────┐ │
│ │ VocabularyListPanel│ │ VocabularyBadge │ │
│ │ - Refreshes list │ │ - Updates count │ │
│ │ - Object pooling │ │ - Shows/hides │ │
│ └────────────────────┘ └──────────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
┌───────────────────────────────────────────────────────────────────────┐
│ LANGUAGE SETTINGS SYSTEM │
├───────────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ LanguageSettings.cs (ScriptableObject) │ │
│ │ │ │
│ │ Properties: │ │
│ │ - CurrentLanguage (Target: e.g., Italian) │ │
│ │ - CurrentSourceLanguage (Source: e.g., French) │ │
│ │ │ │
│ │ Events: │ │
│ │ - OnLanguageChanged │ │
│ │ - OnSourceLanguageChanged │ │
│ │ │ │
│ │ Persistence: │ │
│ │ - PlayerPrefs: "LinguaAR_TargetLanguage" │ │
│ │ - PlayerPrefs: "LinguaAR_SourceLanguage" │ │
│ │ │ │
│ │ Static Helpers: │ │
│ │ - GetLanguageDisplayName() → "Français" │ │
│ │ - GetLanguageFlag() → "🇫🇷" │ │
│ │ - GetLanguageCode() → "fr" │ │
│ └──────────────────┬───────────────────────────────────────┘ │
│ │ │
│ ┌───────────┴──────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌────────────────────┐ │
│ │ Main Menu │ │ AR Scene Uses │ │
│ │ │ │ │ │
│ │- Target │ │- Object Detection │ │
│ │ Dropdown │ │- Translation │ │
│ │- Source │ │- Examples Panel │ │
│ │ Dropdown │ │- Vocabulary Save │ │
│ │- Validation │ └────────────────────┘ │
│ └──────────────┘ │
│ │
└────────────────────────────────────────────────────────────────────────┘
Camera Feed
│
▼
ARObjectDetectionManager (Lightship)
│
├─► Detects: "chair" (confidence: 0.85)
│
▼
ObjectDetectionSample.cs
│
├─► Filters: confidence >= 0.6? ✓
│
├─► Fetches Translation (Source):
│ TranslationDatabase.GetTranslation("chair", French)
│ → "la chaise"
│
├─► Fetches Translation (Target):
│ TranslationDatabase.GetTranslation("chair", Italian)
│ → "la sedia"
│
▼
DrawRect.CreateRect(...)
│
├─► Creates UIRectObject with:
│ - Category: "chair"
│ - SourceTranslation: "la chaise"
│ - Translation: "la sedia"
│ - Confidence: 0.85
│
▼
Screen displays: "la chaise" label over detected chair
User taps detection rectangle
│
▼
UIRectObject.OnPointerClick()
│
▼
ObjectSelectionManager.FocusObject(rect)
│
├─► Creates DetectedObjectData:
│ {
│ category: "chair",
│ translation: "la sedia",
│ sourceTranslation: "la chaise",
│ confidence: 0.85
│ }
│
├─► Fires: OnObjectFocused event
│
▼
ActionMenuPanel.ShowMenu(data)
│
├─► Displays: "la chaise → la sedia"
│
├─► Shows 4 buttons:
│ [🔊 Pronunciation] [📝 Examples] [💾 Save] [❌ Close]
│
▼
User clicks "Save Word"
│
▼
ActionMenuPanel.OnSaveWordClicked()
│
▼
VocabularyManager.SaveWord(
"chair",
"la sedia",
Italian,
"la chaise",
French
)
│
├─► Creates/Updates SavedWord:
│ {
│ category: "chair",
│ savedTranslation: "la sedia",
│ savedLanguage: Italian,
│ sourceTranslation: "la chaise",
│ sourceLanguage: French,
│ firstDetectedTime: "2025-01-15T10:30:00",
│ lastSeenTime: "2025-01-15T10:30:00",
│ timesDetected: 1
│ }
│
├─► Writes to: vocabulary.json
│
├─► Fires: OnVocabularyUpdated event
│
▼
VocabularyBadge.UpdateBadge()
│
└─► Updates badge count: "1"
User changes language in Main Menu
│
▼
LanguageSelectionUI.OnDropdownValueChanged()
│
├─► Validation: Source ≠ Target? ✓
│
▼
LanguageSettings.CurrentSourceLanguage = French
│
├─► Saves to PlayerPrefs
│
├─► Fires: OnSourceLanguageChanged event
│
▼
All Subscribers React:
│
├─► ObjectDetectionSample:
│ - Fetches new source translations
│ - Updates detection labels
│
├─► ExampleSentencesPanel:
│ - Displays examples in French
│
├─► ActionMenuPanel:
│ - Shows "French → Italian" format
│
└─► VocabularyManager:
- Saves words with French source
| Component | Responsibility | Key Methods |
|---|---|---|
| ARObjectDetectionManager | Hardware camera access, ML inference | (Niantic Lightship API) |
| ObjectDetectionSample | Detection controller, filtering, translation | OnFrameReceived(), Pause(), Resume() |
| DrawRect | UI visualization, object pooling | CreateRect(), ClearRects() |
| UIRectObject | Clickable detection rectangle | OnPointerClick(), SetDetectionData() |
| Depth_ScreenToWorldPosition | 2D→3D coordinate conversion | GetWorldPosition() |
| Component | Responsibility | Key Methods |
|---|---|---|
| ObjectSelectionManager | Focus management, event emission | FocusObject(), UnfocusObject() |
| ActionMenuPanel | Action menu UI, button handlers | ShowMenu(), Hide(), OnSaveWordClicked() |
| ExampleSentencesPanel | Example sentences display | ShowExamples(), CreateSentenceItem() |
| TTSManager | Text-to-speech (platform-specific) | Speak(), Stop() |
| Component | Responsibility | Key Methods |
|---|---|---|
| VocabularyManager | CRUD operations, event emission | SaveWord(), DeleteWord(), GetAllWords() |
| VocabularyListPanel | List UI, object pooling, search | RefreshList(), Show(), Hide() |
| VocabularyWordItem | List item UI component | Setup(), OnDeleteClicked() |
| VocabularyBadge | Word count indicator | UpdateBadge() |
| Component | Responsibility | Key Methods |
|---|---|---|
| LanguageSettings | Global language state, persistence | Properties: CurrentLanguage, CurrentSourceLanguage |
| TranslationDatabaseSO | Offline translation lookup | GetTranslation(), GetAllWords() |
| ExampleSentencesDatabaseSO | Example sentences storage | GetExamples(), GetTranslation() |
| LanguageSelectionUI | Language selection UI, validation | SetLanguage(), SetSourceLanguage() |
Used for global access to managers:
public class VocabularyManager : MonoBehaviour
{
public static VocabularyManager Instance { get; private set; }
}Components: VocabularyManager, ExampleSentencesPanel, ObjectSelectionManager, SpawnObjectsAroundObjectDetected
Decouples components via events:
public static event Action OnVocabularyUpdated;
public static event Action<TargetLanguage> OnLanguageChanged;Events:
OnVocabularyUpdated→ VocabularyBadge, VocabularyListPanelOnLanguageChanged→ ObjectDetectionSample, UI componentsOnObjectFocused→ ActionMenuPanel
Reduces GC pressure for frequently created/destroyed objects:
// DrawRect maintains pool of UIRectObject instances
private List<GameObject> rects = new List<GameObject>();Components: DrawRect, VocabularyListPanel
Data-driven design for game data:
[CreateAssetMenu(fileName = "TranslationDB", menuName = "...")]
public class TranslationDatabaseSO : ScriptableObjectAssets: LanguageSettings, TranslationDatabaseSO, ExampleSentencesDatabaseSO, SpawnObjectToObjectClassSO
Platform-specific implementations:
#if UNITY_IOS
private IOSTTSManager _tts;
#elif UNITY_ANDROID
private AndroidTTSManager _tts;
#endifComponents: TTSManager with iOS/Android variants
- Unity: 6000.2.10f1 (Unity 6)
- Render Pipeline: Universal Render Pipeline (URP)
- AR Foundation: Cross-platform AR abstraction
- Niantic Lightship ARDK: Advanced AR features
- Object detection (80+ categories)
- Semantic segmentation
- Depth estimation
- Persistent anchors
- iOS: ARKit + Lightship ARKit Loader
- Android: ARCore + Lightship ARCore Loader
- VR: OpenXR Loader
- Editor: Lightship Simulation + XR Environment Simulation
- Translation: Offline TranslationDatabaseSO
- TTS: Native iOS/Android speech synthesis
- Persistence: JSON (JsonUtility) + PlayerPrefs
- TextMeshPro (UI text rendering)
- Newtonsoft.Json (JSON serialization)
- Input System (new input handling)
- Unity 6000.2.10f1 (exact version required)
- Unity Hub recommended
- Platform modules:
- iOS Build Support (for iOS builds)
- Android Build Support (for Android builds)
- OpenXR Plugin (for VR)
-
Clone Repository
git clone <repository-url> cd "Lingua AR"
-
Open in Unity Hub
- Open Unity Hub
- Click "Add" → Select project folder
- Unity version: 6000.2.10f1
- Click "Open"
-
Package Resolution
- Packages auto-resolve from
Packages/manifest.json - Wait for compilation to complete
- Packages auto-resolve from
-
Niantic Lightship Setup
- Obtain API key from Niantic Lightship
- Open
Assets/Settings/LightshipSettings.asset - Paste API key
-
Test in Editor
- Open
Assets/Scenes/AR Scene.unity - Enter Play mode
- XR Simulation activates automatically
- Open
-
Load AR Scene
- Open
Assets/Scenes/AR Scene.unity
- Open
-
Select Simulation Environment
- Window → XR → AR Foundation → XR Environment
- Choose: Kitchen, Living Room, Office, etc.
-
Controls
- Mouse: Look around
- WASD: Move camera
- Right-click + drag: Rotate view
-
Interact
- Detection rectangles appear on objects
- Click rectangle to open action menu
- Test pronunciation, examples, vocabulary save