feat: add multimodal benchmark history and observability - #1
Conversation
Sample target process trees, cgroups, host CPU and memory, NVML devices, Ethernet, and RDMA with source timestamps, bounded retries, and an acknowledged final flush. Signed-off-by: ActivePeter <1020401660@qq.com>
Add versioned target mappings, lifecycle correlation, normalized artifacts, and standalone Prometheus collection while retaining raw implementation-specific evidence. Signed-off-by: ActivePeter <1020401660@qq.com>
Profile TeleFuser WebRTC and SGLang MessagePack WebSocket targets through typed contracts, shared session timelines, warmup-aware target measurements, artifacts, reports, and optional active resources. Signed-off-by: ActivePeter <1020401660@qq.com>
Bundle a Chinese desktop Vue and ECharts interface with two run groups, canonical metric-tree selection, compact charts, top-layer tooltips, and resource usage/capacity views. Signed-off-by: ActivePeter <1020401660@qq.com>
Index profile and stream artifacts, ingest idempotent live resource batches, expose bounded history APIs, and serve the bundled dashboard with GreptimeDB as the required backend. Signed-off-by: ActivePeter <1020401660@qq.com>
Document stream contracts, canonical metrics, active resources, GreptimeDB operations, dashboard usage, failure semantics, and generated CLI options. Signed-off-by: ActivePeter <1020401660@qq.com>
Use pytest-provided ephemeral ports so parallel unit runs do not collide with long-lived services on shared hosts. Signed-off-by: ActivePeter <1020401660@qq.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive benchmark history service backed by GreptimeDB, featuring a new CLI for data ingestion and a Vue-based dashboard for visualization. It also implements multimodal benchmark extensions and active resource telemetry collection for monitoring process trees, cgroups, and GPU usage. The reviewer provided actionable feedback, including suggestions to prevent silent data overwrites in the video generation endpoint, improve type safety in the payload merging logic, and add logging for potential PID truncation in the cgroup collector.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if turn.images and turn.images[0].contents: | ||
| image_content = turn.images[0].contents[0] | ||
| if image_content.lower().startswith(("http://", "https://")): | ||
| payload["reference_url"] = image_content | ||
| else: | ||
| payload["input_reference"] = ImageEditEndpoint._build_image_field( | ||
| image_content | ||
| ) | ||
|
|
||
| if turn.videos and turn.videos[0].contents: | ||
| video_content = turn.videos[0].contents[0] | ||
| if video_content.lower().startswith(("http://", "https://")): | ||
| payload["reference_url"] = video_content | ||
| else: | ||
| payload["input_reference"] = self._build_video_field(video_content) |
There was a problem hiding this comment.
If both an image and a video are provided in the turn, the video data will overwrite the image data in the payload because both branches can set reference_url and input_reference. If the API doesn't support both simultaneously, it might be better to raise an error or log a warning to avoid this silent overwrite and make the behavior more explicit to the user.
| def _merge_with_reserved_filter( | ||
| self, | ||
| payload: dict[str, Any], | ||
| items: Any, | ||
| label: str, | ||
| ) -> None: |
There was a problem hiding this comment.
For better type safety and code clarity, the items parameter should be typed more specifically as Iterable[tuple[str, Any]] instead of Any. This more accurately reflects its usage with both a list of tuples and dictionary items.
You'll need to add from collections.abc import Iterable at the top of the file.
| def _merge_with_reserved_filter( | |
| self, | |
| payload: dict[str, Any], | |
| items: Any, | |
| label: str, | |
| ) -> None: | |
| def _merge_with_reserved_filter( | |
| self, | |
| payload: dict[str, Any], | |
| items: Iterable[tuple[str, Any]], | |
| label: str, | |
| ) -> None: |
| process_ids.update(values) | ||
| if len(process_ids) >= _MAX_CGROUP_PROCESSES: |
There was a problem hiding this comment.
If the number of process IDs found in the cgroup subtree exceeds _MAX_CGROUP_PROCESSES, the list of PIDs is truncated. This could lead to incomplete resource attribution for the container without any warning. Consider logging a warning when this truncation occurs to alert the user that the collected container resource metrics might be incomplete.
|
Follow-up hardening landed on |
Summary
Validation
Integration
The companion TeleFuser PR fixes the AIPerf checkout at
<TeleFuser>/benchmarks/aiperfand contains only thin target launchers, contracts, and raw runtime facts.Companion TeleFuser PR: Tele-AI/TeleFuser#13