Skip to content

feat: add multimodal benchmark history and observability - #1

Merged
ActivePeter merged 7 commits into
teleaifrom
activepeter/multimodal-history
Jul 21, 2026
Merged

feat: add multimodal benchmark history and observability#1
ActivePeter merged 7 commits into
teleaifrom
activepeter/multimodal-history

Conversation

@ActivePeter

@ActivePeter ActivePeter commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • add reusable stream-world profiling for TeleFuser and SGLang targets
  • add active process-tree, host, container, GPU, Ethernet, and RDMA telemetry
  • add required GreptimeDB history ingestion and a Vue 3 plus TypeScript comparison dashboard
  • group the UI by canonical design dimensions with multi-run left and right comparison
  • document the multimodal contracts, metrics, history service, and target integration boundary

Validation

  • 13,640 unit tests passed, 98 skipped
  • full pre-commit suite passed
  • Vue TypeScript production build passed with no generated-artifact drift
  • all stream configs and contracts loaded through their real typed parsers
  • live 2xH100 TeleFuser profile persisted 4,273 merged statistic and resource points to GreptimeDB with zero dropped samples

Integration

The companion TeleFuser PR fixes the AIPerf checkout at <TeleFuser>/benchmarks/aiperf and contains only thin target launchers, contracts, and raw runtime facts.

Companion TeleFuser PR: Tele-AI/TeleFuser#13

Sample target process trees, cgroups, host CPU and memory, NVML devices, Ethernet, and RDMA with source timestamps, bounded retries, and an acknowledged final flush.

Signed-off-by: ActivePeter <1020401660@qq.com>
Add versioned target mappings, lifecycle correlation, normalized artifacts, and standalone Prometheus collection while retaining raw implementation-specific evidence.

Signed-off-by: ActivePeter <1020401660@qq.com>
Profile TeleFuser WebRTC and SGLang MessagePack WebSocket targets through typed contracts, shared session timelines, warmup-aware target measurements, artifacts, reports, and optional active resources.

Signed-off-by: ActivePeter <1020401660@qq.com>
Bundle a Chinese desktop Vue and ECharts interface with two run groups, canonical metric-tree selection, compact charts, top-layer tooltips, and resource usage/capacity views.

Signed-off-by: ActivePeter <1020401660@qq.com>
Index profile and stream artifacts, ingest idempotent live resource batches, expose bounded history APIs, and serve the bundled dashboard with GreptimeDB as the required backend.

Signed-off-by: ActivePeter <1020401660@qq.com>
Document stream contracts, canonical metrics, active resources, GreptimeDB operations, dashboard usage, failure semantics, and generated CLI options.

Signed-off-by: ActivePeter <1020401660@qq.com>
Use pytest-provided ephemeral ports so parallel unit runs do not collide with long-lived services on shared hosts.

Signed-off-by: ActivePeter <1020401660@qq.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive benchmark history service backed by GreptimeDB, featuring a new CLI for data ingestion and a Vue-based dashboard for visualization. It also implements multimodal benchmark extensions and active resource telemetry collection for monitoring process trees, cgroups, and GPU usage. The reviewer provided actionable feedback, including suggestions to prevent silent data overwrites in the video generation endpoint, improve type safety in the payload merging logic, and add logging for potential PID truncation in the cgroup collector.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +73 to +87
if turn.images and turn.images[0].contents:
image_content = turn.images[0].contents[0]
if image_content.lower().startswith(("http://", "https://")):
payload["reference_url"] = image_content
else:
payload["input_reference"] = ImageEditEndpoint._build_image_field(
image_content
)

if turn.videos and turn.videos[0].contents:
video_content = turn.videos[0].contents[0]
if video_content.lower().startswith(("http://", "https://")):
payload["reference_url"] = video_content
else:
payload["input_reference"] = self._build_video_field(video_content)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If both an image and a video are provided in the turn, the video data will overwrite the image data in the payload because both branches can set reference_url and input_reference. If the API doesn't support both simultaneously, it might be better to raise an error or log a warning to avoid this silent overwrite and make the behavior more explicit to the user.

Comment on lines +99 to +104
def _merge_with_reserved_filter(
self,
payload: dict[str, Any],
items: Any,
label: str,
) -> None:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For better type safety and code clarity, the items parameter should be typed more specifically as Iterable[tuple[str, Any]] instead of Any. This more accurately reflects its usage with both a list of tuples and dictionary items.

You'll need to add from collections.abc import Iterable at the top of the file.

Suggested change
def _merge_with_reserved_filter(
self,
payload: dict[str, Any],
items: Any,
label: str,
) -> None:
def _merge_with_reserved_filter(
self,
payload: dict[str, Any],
items: Iterable[tuple[str, Any]],
label: str,
) -> None:

Comment on lines +469 to +470
process_ids.update(values)
if len(process_ids) >= _MAX_CGROUP_PROCESSES:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

If the number of process IDs found in the cgroup subtree exceeds _MAX_CGROUP_PROCESSES, the list of PIDs is truncated. This could lead to incomplete resource attribution for the container without any warning. Consider logging a warning when this truncation occurs to alert the user that the collected container resource metrics might be incomplete.

@ActivePeter
ActivePeter merged commit 373e142 into teleai Jul 21, 2026
@ActivePeter

ActivePeter commented Jul 21, 2026

Copy link
Copy Markdown
Owner Author

Follow-up hardening landed on teleai in f5e936e9, with the fresh-checkout artifact-root documentation correction in e977ffbb: runtime setup guidance, image/video reference validation, and bounded cgroup PID traversal that disables partial container-GPU attribution when scan limits are exceeded. Targeted tests, pre-commit, ergonomics checks, clean-room History startup, real-artifact ingest, and wheel asset verification passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant