Skip to content

Add built-in on-device vision, language, and inference#5467

Open
shai-almog wants to merge 10 commits into
masterfrom
builtin-on-device-ai
Open

Add built-in on-device vision, language, and inference#5467
shai-almog wants to merge 10 commits into
masterfrom
builtin-on-device-ai

Conversation

@shai-almog

Copy link
Copy Markdown
Collaborator

Summary

  • add vendor-neutral built-in APIs for on-device vision, language, and LiteRT/Core ML inference
  • add Android ML Kit/LiteRT implementations, iOS Vision/NaturalLanguage/Core ML implementations, and Mac Catalyst Vision support
  • centralize granular class- and method-level source/dependency selection in PlatformFeatureCatalog, shared with BuildDaemon
  • retire the small ML Kit and TFLite cn1libs while retaining large-model libraries such as Whisper and Stable Diffusion
  • document API availability, build hints, migration, and platform limitations
  • add focused core/catalog/builder tests plus HelloCodenameOne and Port Status coverage without screenshot requirements

Validation

  • Java 8 core compile: 1,615 sources
  • focused core API tests: 9 passed
  • platform feature catalog tests: 20 passed
  • focused builder selection tests: 24 passed before rebasing; the only rebase conflict was in Port Status coverage
  • Port Status validator: 170 tests, 57 features, 11 ports, 303 golden names
  • Port Status conformance tests: 9 passed
  • HelloCodenameOne shared-source Java compile: 196 sources
  • Android adapters compiled against the official ML Kit and LiteRT dependencies
  • iOS all-feature device Xcode build passed
  • Mac Catalyst default Apple Vision build passed

Platform note

Google ML Kit's MLImage dependency currently contains a device-only slice, so an arm64 iOS Simulator build that explicitly selects the ML Kit vision backend cannot link. Device builds pass, and the default Apple Vision path does not pull that pod into simulator builds.

Copilot AI review requested due to automatic review settings July 25, 2026 11:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings July 25, 2026 11:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@github-actions

Copy link
Copy Markdown
Contributor

Cloudflare Preview

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Developer Guide build artifacts are available for download from this workflow run:

Developer Guide quality checks:

  • AsciiDoc linter: No issues found (report)
  • Vale: No alerts found (report)
  • Paragraph capitalization: No paragraph capitalization issues (report)
  • LanguageTool: No grammar matches (report)
  • Image references: No unused images detected (report)

Copilot AI review requested due to automatic review settings July 25, 2026 12:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings July 25, 2026 12:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings July 25, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 151 screenshots: 151 matched.

Native Android coverage

  • 📊 Line coverage: 7.73% (7476/96721 lines covered) [HTML preview] (artifact android-coverage-report, jacocoAndroidReport/html/index.html)
    • Other counters: instruction 7.61% (39007/512565), branch 2.74% (1333/48583), complexity 3.07% (1600/52127), method 4.73% (1303/27567), class 9.61% (353/3672)
    • Lowest covered classes
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysKt – 0.00% (0/6367 lines covered)
      • kotlin.collections.unsigned.kotlin.collections.unsigned.UArraysKt___UArraysKt – 0.00% (0/2384 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.ClassReader – 0.00% (0/1524 lines covered)
      • kotlin.collections.kotlin.collections.CollectionsKt___CollectionsKt – 0.00% (0/1187 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.MethodWriter – 0.00% (0/922 lines covered)
      • kotlin.sequences.kotlin.sequences.SequencesKt___SequencesKt – 0.00% (0/736 lines covered)
      • com.google.common.cache.com.google.common.cache.LocalCache$Segment – 0.00% (0/726 lines covered)
      • okio.okio.Buffer – 0.00% (0/687 lines covered)
      • kotlin.text.kotlin.text.StringsKt___StringsKt – 0.00% (0/625 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.Frame – 0.00% (0/570 lines covered)

✅ Native Android screenshot tests passed.

Native Android coverage

  • 📊 Line coverage: 7.73% (7476/96721 lines covered) [HTML preview] (artifact android-coverage-report, jacocoAndroidReport/html/index.html)
    • Other counters: instruction 7.61% (39007/512565), branch 2.74% (1333/48583), complexity 3.07% (1600/52127), method 4.73% (1303/27567), class 9.61% (353/3672)
    • Lowest covered classes
      • kotlin.collections.kotlin.collections.ArraysKt___ArraysKt – 0.00% (0/6367 lines covered)
      • kotlin.collections.unsigned.kotlin.collections.unsigned.UArraysKt___UArraysKt – 0.00% (0/2384 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.ClassReader – 0.00% (0/1524 lines covered)
      • kotlin.collections.kotlin.collections.CollectionsKt___CollectionsKt – 0.00% (0/1187 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.MethodWriter – 0.00% (0/922 lines covered)
      • kotlin.sequences.kotlin.sequences.SequencesKt___SequencesKt – 0.00% (0/736 lines covered)
      • com.google.common.cache.com.google.common.cache.LocalCache$Segment – 0.00% (0/726 lines covered)
      • okio.okio.Buffer – 0.00% (0/687 lines covered)
      • kotlin.text.kotlin.text.StringsKt___StringsKt – 0.00% (0/625 lines covered)
      • org.jacoco.agent.rt.internal_0e20598.asm.org.jacoco.agent.rt.internal_0e20598.asm.Frame – 0.00% (0/570 lines covered)

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend scalar fallback (no native SIMD)
SIMD int-add (64K x300) java 300ms / native 61ms = 4.9x speedup
SIMD float-mul (64K x300) java 174ms / native 122ms = 1.4x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 97.000 ms
Base64 CN1 decode 92.000 ms
Base64 native encode 333.000 ms
Base64 encode ratio (CN1/native) 0.291x (70.9% faster)
Base64 native decode 307.000 ms
Base64 decode ratio (CN1/native) 0.300x (70.0% faster)
Image encode benchmark status skipped (SIMD unsupported)

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 144 screenshots: 144 matched.
✅ Native Apple TV (tvOS, Metal) screenshot tests passed.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 146 screenshots: 146 matched.
Native Windows port (x64 / Intel-AMD): full hellocodenameone screenshot suite rendered offscreen with Direct2D/DirectWrite, plus the real benchmarks (base64 native/CN1/SIMD, image createMask/applyMask/modifyAlpha/PNG/JPEG, SSE2 SIMD kernels). Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 73ms / native 5ms = 14.6x speedup
SIMD float-mul (64K x300) java 62ms / native 5ms = 12.4x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 195.000 ms
Base64 CN1 decode 156.000 ms
Base64 SIMD encode 101.000 ms
Base64 encode ratio (SIMD/CN1) 0.518x (48.2% faster)
Base64 SIMD decode 83.000 ms
Base64 decode ratio (SIMD/CN1) 0.532x (46.8% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 26.000 ms
Image createMask (SIMD on) 107.000 ms
Image createMask ratio (SIMD on/off) 4.115x (311.5% slower)
Image applyMask (SIMD off) 63.000 ms
Image applyMask (SIMD on) 56.000 ms
Image applyMask ratio (SIMD on/off) 0.889x (11.1% faster)
Image modifyAlpha (SIMD off) 59.000 ms
Image modifyAlpha (SIMD on) 54.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.915x (8.5% faster)
Image modifyAlpha removeColor (SIMD off) 41.000 ms
Image modifyAlpha removeColor (SIMD on) 29.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.707x (29.3% faster)

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 146 screenshots: 146 matched.
Native Windows port, REAL shipping pipeline: the hellocodenameone screenshot suite rendered by a binary CROSS-COMPILED on Linux (clang-cl + xwin, WebView2 linked) and RUN on a Windows x64 runner. Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 55ms / native 4ms = 13.7x speedup
SIMD float-mul (64K x300) java 55ms / native 4ms = 13.7x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 174.000 ms
Base64 CN1 decode 101.000 ms
Base64 SIMD encode 87.000 ms
Base64 encode ratio (SIMD/CN1) 0.500x (50.0% faster)
Base64 SIMD decode 76.000 ms
Base64 decode ratio (SIMD/CN1) 0.752x (24.8% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 139.000 ms
Image createMask (SIMD on) 19.000 ms
Image createMask ratio (SIMD on/off) 0.137x (86.3% faster)
Image applyMask (SIMD off) 48.000 ms
Image applyMask (SIMD on) 39.000 ms
Image applyMask ratio (SIMD on/off) 0.813x (18.8% faster)
Image modifyAlpha (SIMD off) 37.000 ms
Image modifyAlpha (SIMD on) 31.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.838x (16.2% faster)
Image modifyAlpha removeColor (SIMD off) 36.000 ms
Image modifyAlpha removeColor (SIMD on) 30.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.833x (16.7% faster)

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 146 screenshots: 146 matched.
Native Windows port (arm64 / Apple Silicon - Arm): full hellocodenameone screenshot suite rendered offscreen with Direct2D/DirectWrite, plus the real benchmarks (base64 native/CN1/SIMD, image createMask/applyMask/modifyAlpha/PNG/JPEG, NEON SIMD kernels). Compared against the in-repo baseline in scripts/windows/screenshots.

Benchmark Results

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 56ms / native 3ms = 18.6x speedup
SIMD float-mul (64K x300) java 55ms / native 3ms = 18.3x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 native bridge unavailable (CN1 + SIMD + image benchmarks only)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 247.000 ms
Base64 CN1 decode 127.000 ms
Base64 SIMD encode 65.000 ms
Base64 encode ratio (SIMD/CN1) 0.263x (73.7% faster)
Base64 SIMD decode 59.000 ms
Base64 decode ratio (SIMD/CN1) 0.465x (53.5% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 11.000 ms
Image createMask (SIMD on) 9.000 ms
Image createMask ratio (SIMD on/off) 0.818x (18.2% faster)
Image applyMask (SIMD off) 24.000 ms
Image applyMask (SIMD on) 19.000 ms
Image applyMask ratio (SIMD on/off) 0.792x (20.8% faster)
Image modifyAlpha (SIMD off) 16.000 ms
Image modifyAlpha (SIMD on) 12.000 ms
Image modifyAlpha ratio (SIMD on/off) 0.750x (25.0% faster)
Image modifyAlpha removeColor (SIMD off) 18.000 ms
Image modifyAlpha removeColor (SIMD on) 12.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.667x (33.3% faster)

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 181 screenshots: 181 matched.
✅ JavaScript-port screenshot tests passed.

Copilot AI review requested due to automatic review settings July 25, 2026 13:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

✅ Continuous Quality Report

Test & Coverage

Static Analysis

  • SpotBugs [HTML preview] [Download]
    • ByteCodeTranslator: 0 findings (no issues)
    • android: 1 findings (Normal: 1)
    • codenameone-maven-plugin: 0 findings (no issues)
    • core-unittests: 0 findings (no issues)
    • ios: 0 findings (no issues)
  • PMD: 0 findings (no issues) [Report archive]
  • Checkstyle: 0 findings (no issues) [Report archive]

Generated automatically by the PR CI workflow.

Copilot AI review requested due to automatic review settings July 25, 2026 13:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 217 screenshots: 217 matched.
✅ Native Apple Watch (watchOS, Core Graphics) screenshot tests passed.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 147 screenshots: 147 matched.
Native Linux port (x64), GTK3/Cairo/Pango, ParparVM bytecode-to-C (no JVM): the hellocodenameone screenshot suite rendered by a native ELF built + run on the GitHub x64 runner. Baseline: scripts/linux/screenshots.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 147 screenshots: 147 matched.
Native Linux port (arm64), GTK3/Cairo/Pango, ParparVM bytecode-to-C (no JVM): the hellocodenameone screenshot suite rendered by a native ELF built + run on the GitHub arm64 runner. Baseline: scripts/linux/screenshots-arm.

Copilot AI review requested due to automatic review settings July 25, 2026 15:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

Compared 148 screenshots: 148 matched.
✅ Native Mac screenshot tests passed.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 197 seconds

Detailed Performance Metrics

Metric Duration
SIMD kernel backend SSE2 (x64) / NEON (arm64) native kernels
SIMD int-add (64K x300) java 54ms / native 3ms = 18.0x speedup
SIMD float-mul (64K x300) java 54ms / native 3ms = 18.0x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path active (NEON-accelerated)
Base64 CN1 encode 155.000 ms
Base64 CN1 decode 114.000 ms
Base64 native encode 458.000 ms
Base64 encode ratio (CN1/native) 0.338x (66.2% faster)
Base64 native decode 195.000 ms
Base64 decode ratio (CN1/native) 0.585x (41.5% faster)
Base64 SIMD encode 48.000 ms
Base64 encode ratio (SIMD/CN1) 0.310x (69.0% faster)
Base64 SIMD decode 49.000 ms
Base64 decode ratio (SIMD/CN1) 0.430x (57.0% faster)
Base64 encode ratio (SIMD/native) 0.105x (89.5% faster)
Base64 decode ratio (SIMD/native) 0.251x (74.9% faster)
Image encode benchmark iterations 100
Image createMask (SIMD off) 7.000 ms
Image createMask (SIMD on) 1.000 ms
Image createMask ratio (SIMD on/off) 0.143x (85.7% faster)
Image applyMask (SIMD off) 41.000 ms
Image applyMask (SIMD on) 29.000 ms
Image applyMask ratio (SIMD on/off) 0.707x (29.3% faster)
Image modifyAlpha (SIMD off) 32.000 ms
Image modifyAlpha (SIMD on) 32.000 ms
Image modifyAlpha ratio (SIMD on/off) 1.000x (0.0% slower)
Image modifyAlpha removeColor (SIMD off) 35.000 ms
Image modifyAlpha removeColor (SIMD on) 30.000 ms
Image modifyAlpha removeColor ratio (SIMD on/off) 0.857x (14.3% faster)

Copilot AI review requested due to automatic review settings July 25, 2026 17:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

Copilot AI review requested due to automatic review settings July 25, 2026 17:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@shai-almog

shai-almog commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator Author

iOS screenshot updates

Compared 143 screenshots: 142 matched, 1 updated.

  • FloatingActionButtonTheme_light — updated screenshot. Screenshot differs (1179x2556 px, bit depth 8).

    FloatingActionButtonTheme_light
    Preview info: JPEG preview quality 70; JPEG preview quality 70; downscaled to 825x1789.
    Full-resolution PNG saved as FloatingActionButtonTheme_light.png in workflow artifacts.

Benchmark Results

  • VM Translation Time: 0 seconds
  • Compilation Time: 259 seconds

Build and Run Timing

Metric Duration
Simulator Boot 54000 ms
Simulator Boot (Run) 0 ms
App Install 10000 ms
App Launch 3000 ms
Test Execution 1431000 ms

Detailed Performance Metrics

Metric Duration
SIMD kernel backend scalar fallback (no native SIMD)
SIMD int-add (64K x300) java 72ms / native 46ms = 1.5x speedup
SIMD float-mul (64K x300) java 77ms / native 61ms = 1.2x speedup
SIMD kernel correctness PASS (native result == scalar reference)
Base64 payload size 8192 bytes
Base64 benchmark iterations 6000
Base64 SIMD byte path gated to scalar (CPU autovectorizes scalar; explicit SIMD not beneficial here)
Base64 CN1 encode 203.000 ms
Base64 CN1 decode 151.000 ms
Base64 native encode 1578.000 ms
Base64 encode ratio (CN1/native) 0.129x (87.1% faster)
Base64 native decode 1289.000 ms
Base64 decode ratio (CN1/native) 0.117x (88.3% faster)
Base64 SIMD benchmark status unsupported on this runtime
Image encode benchmark status skipped (SIMD unsupported)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants