Add MPQ/deploy dashboard history export, visualization, and live search plotting#39
Conversation
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fbf69a10-c3e7-4943-bbf1-0ad748125836 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fbf69a10-c3e7-4943-bbf1-0ad748125836 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fbf69a10-c3e7-4943-bbf1-0ad748125836 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fbf69a10-c3e7-4943-bbf1-0ad748125836 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fbf69a10-c3e7-4943-bbf1-0ad748125836 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Reviewer's GuideIntroduces a reusable MiCoDashboard utility and CLI to export, aggregate, and visualize MPQ search/deploy history (accuracy vs constraint), and refactors all searchers to track both best objective values and corresponding schemes in a consistent way that can be serialized for dashboard consumption. Sequence diagram for MPQ search with dashboard history exportsequenceDiagram
actor User
participant MPQSearchScript as mpq_search_py
participant Searcher as QSearcher_subclass
participant Evaluator as MiCoEval
participant Dashboard as MiCoDashboard
participant FS as FileSystem
User->>MPQSearchScript: run mpq_search_py(model_name, mode, constr_ratio)
MPQSearchScript->>Searcher: search(n_iter, target, constr, constr_value)
activate Searcher
Searcher->>Searcher: start_search(target, constr, constr_value)
Searcher->>Evaluator: set_eval(target)
alt has_constraint
Searcher->>Evaluator: set_constraint(constr)
end
loop search_iterations
Searcher->>Evaluator: evaluate(candidate_scheme)
Evaluator-->>Searcher: metric_value
Searcher->>Searcher: record_best(best_scheme, best_value)
note over Searcher: Append to best_trace
note over Searcher: Append to best_scheme_trace
end
Searcher-->>MPQSearchScript: best_scheme, best_value
deactivate Searcher
MPQSearchScript->>Dashboard: build_run_history(Searcher, Evaluator, constraint_name)
activate Dashboard
Dashboard->>Evaluator: eval_dict()
Evaluator-->>Dashboard: eval_map
loop over best_trace
Dashboard->>Evaluator: constr_eval(scheme)
Evaluator-->>Dashboard: constr_value
Dashboard->>Dashboard: append {iter, accuracy, constraint, scheme}
end
Dashboard-->>MPQSearchScript: history
deactivate Dashboard
MPQSearchScript->>Dashboard: build_run_entry(method, seed, objective, constraint_name, constraint_limit, history)
Dashboard-->>MPQSearchScript: run_entry
MPQSearchScript->>Dashboard: save_runs(history_json_path, [run_entry,...])
Dashboard->>FS: write_json(path, {runs: [...]})
FS-->>Dashboard: ok
Dashboard-->>MPQSearchScript: done
MPQSearchScript-->>User: print("Dashboard history JSON saved")
ER diagram for dashboard JSON schema (runs and history)erDiagram
RUN {
string method
int seed
string objective
string constraint_name
float constraint_limit
}
HISTORY_POINT {
int iter
float accuracy
float constraint
string scheme
}
DASHBOARD_FILE {
json runs
}
RUN ||--o{ HISTORY_POINT : has_history
DASHBOARD_FILE ||--o{ RUN : contains_runs
Class diagram for updated searchers and MiCoDashboardclassDiagram
class QSearcher {
+int n_inits
+list qtypes
+list best_trace
+list best_scheme_trace
+str target
+str constr_name
+float constr_value
+__init__(MiCoEval evaluator, int n_inits, list qtypes)
+start_search(str target, str constr, float constr_value)
+record_best(list best_scheme, float best_value)
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class BayesSearcher {
+list sampled_X
+list sampled_y
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class MiCoSearcher {
+list sampled_X
+list sampled_y
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class RegressionSearcher {
+list sampled_X
+list sampled_y
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class HAQSearcher {
+list best_trace
+float best_acc
+list best_res
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class NLPSearcher {
+list best_trace
+list best_scheme_trace
+search(int n_iter, str target, str constr, float constr_value): Tuple
}
class MiCoEval {
+int n_layers
+int dim
+set_eval(str target)
+set_constraint(str constr)
+dict eval_dict()
}
class MiCoDashboard {
+build_run_history(QSearcher searcher, MiCoEval evaluator, str constraint_name): list
+build_run_entry(str method, int seed, str objective, str constraint_name, float constraint_limit, list history): dict
+save_runs(str path, list runs)
+load_runs(str path): list
+run_label(dict run): str
+plot_acc_vs_constr(list runs, str objective, str constraint, str output_path)
+print_top_configs(list runs, int topk)
}
QSearcher <|-- BayesSearcher
QSearcher <|-- MiCoSearcher
QSearcher <|-- RegressionSearcher
QSearcher <|-- HAQSearcher
QSearcher <|-- NLPSearcher
QSearcher --> MiCoEval : uses
BayesSearcher --> MiCoEval : uses
MiCoSearcher --> MiCoEval : uses
RegressionSearcher --> MiCoEval : uses
HAQSearcher --> MiCoEval : uses
NLPSearcher --> MiCoEval : uses
MiCoDashboard --> QSearcher : reads_best_trace_and_best_scheme_trace
MiCoDashboard --> MiCoEval : uses_eval_dict_for_constraint
Flow diagram for mpq_dashboard CLI visualization pipelineflowchart LR
User([User])
A["Run mpq_dashboard_py with input_json"]
B["MiCoDashboard_load_runs"]
C["MiCoDashboard_plot_acc_vs_constr"]
D["MiCoDashboard_print_top_configs"]
E["Read dashboard JSON file"]
F["Runs list (history per run)"]
G["Generate accuracy vs constraint plot"]
H["Save PNG to output path"]
I["Print top K configurations to console"]
User --> A
A --> B
B --> E
E --> F
F --> C
F --> D
C --> G
G --> H
D --> I
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
@copilot I think it's changing too many loc. Try some abstraction. Get me a Class like |
…ports Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
…raint key Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
|
@gemini-code-assist review |
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/38fa873f-d98a-4863-94e1-fdb0b130f0bc Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a dashboarding system for Mixed Precision Quantization (MPQ) search results. It includes a new MiCoDashboard class for managing and visualizing search history, updates to deployment and search scripts to capture this data, and a new example script for generating plots. Additionally, the searcher classes were refactored to use standardized methods for tracking the best results and their corresponding schemes. Feedback focuses on maintaining synchronization between accuracy and scheme traces in mpq_search.py, ensuring consistent file encoding for JSON operations, and optimizing data processing in the plotting logic.
| @@ -120,6 +123,17 @@ | |||
| print(f"Final QAT Accuracy: {final_acc}") | |||
| searcher.best_trace.append(final_acc) | |||
There was a problem hiding this comment.
Directly appending to searcher.best_trace breaks the synchronization with searcher.best_scheme_trace. This causes the final QAT result to be associated with a None scheme in the dashboard history, which leads to it being excluded from plots in MiCoDashboard.plot_acc_vs_constr. Use searcher.record_best() to keep both traces in sync.
| searcher.best_trace.append(final_acc) | |
| searcher.record_best(res_x, final_acc) |
|
|
||
| @staticmethod | ||
| def save_runs(path: str, runs: list): | ||
| output_dir = os.path.dirname(path) |
There was a problem hiding this comment.
| os.makedirs(output_dir, exist_ok=True) | ||
| with open(path, "w") as f: | ||
| json.dump({"runs": runs}, f, indent=2) | ||
|
|
| color_idx = 0 | ||
| for run in runs: |
There was a problem hiding this comment.
These list comprehensions iterate over the points list twice and perform the same filtering logic. This can be optimized by filtering once and then extracting the values.
valid_points = [p for p in points if p.get("constraint") is not None and p.get("accuracy") is not None]
xs = [p.get("constraint") for p in valid_points]
ys = [p.get("accuracy") for p in valid_points]
Refactored to a single |
|
@copilot address the reviews from gemini-code-assist |
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/045ed786-5a17-4c91-8412-12f13a247916 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Addressed Gemini review items in |
There was a problem hiding this comment.
Hey - I've found 2 issues, and left some high level feedback:
- In
HAQSearcher.search,self.best_resis reset toNoneand passed intorecord_best, but from the shown code it’s never updated when a new best policy is found, which meansbest_scheme_trace(and thus dashboard constraint values) will beNoneunless you explicitly setself.best_reswheneverself.best_accis updated. - In
MiCoDashboard.plot_acc_vs_constr, consider callingplt.close()after saving the figure to avoid accumulating open figures and memory usage when plotting multiple dashboards in the same process.
Prompt for AI Agents
Please address the comments from this code review:
## Overall Comments
- In `HAQSearcher.search`, `self.best_res` is reset to `None` and passed into `record_best`, but from the shown code it’s never updated when a new best policy is found, which means `best_scheme_trace` (and thus dashboard constraint values) will be `None` unless you explicitly set `self.best_res` whenever `self.best_acc` is updated.
- In `MiCoDashboard.plot_acc_vs_constr`, consider calling `plt.close()` after saving the figure to avoid accumulating open figures and memory usage when plotting multiple dashboards in the same process.
## Individual Comments
### Comment 1
<location path="MiCoDashboard.py" line_range="8-27" />
<code_context>
+
+class MiCoDashboard:
+ @staticmethod
+ def build_run_history(searcher, evaluator, constraint_name: str):
+ history = []
+ eval_map = evaluator.eval_dict()
+ if constraint_name not in eval_map:
+ valid_names = ", ".join(sorted(eval_map.keys()))
+ raise ValueError(
+ f"Unsupported constraint name '{constraint_name}'. Must be one of: {valid_names}"
+ )
+ constr_eval = eval_map[constraint_name]
+ for idx, best_acc in enumerate(searcher.best_trace):
+ scheme = searcher.best_scheme_trace[idx] if idx < len(searcher.best_scheme_trace) else None
+ constr_val = constr_eval(scheme) if scheme is not None else None
+ history.append({
+ "iter": idx + 1,
+ "accuracy": float(best_acc) if best_acc is not None else None,
</code_context>
<issue_to_address>
**suggestion:** Guard against mismatched `best_trace` / `best_scheme_trace` lengths more explicitly.
`build_run_history` currently indexes `best_scheme_trace` with `idx` and falls back to `None` when `idx >= len(best_scheme_trace)`. If a `searcher` implementation updates `best_trace` and `best_scheme_trace` at different times, this will silently drop or misalign schemes in the history. Instead of quietly returning `None`, consider asserting equal lengths (or otherwise validating alignment) and logging or raising when they differ so inconsistencies are surfaced early.
```suggestion
@staticmethod
def build_run_history(searcher, evaluator, constraint_name: str):
history = []
eval_map = evaluator.eval_dict()
if constraint_name not in eval_map:
valid_names = ", ".join(sorted(eval_map.keys()))
raise ValueError(
f"Unsupported constraint name '{constraint_name}'. Must be one of: {valid_names}"
)
# Validate that best_trace and best_scheme_trace are aligned
if not hasattr(searcher, "best_trace") or not hasattr(searcher, "best_scheme_trace"):
raise AttributeError(
"Searcher must define both 'best_trace' and 'best_scheme_trace' attributes "
"for MiCoDashboard.build_run_history to work."
)
if len(searcher.best_trace) != len(searcher.best_scheme_trace):
raise ValueError(
"Mismatched lengths for 'best_trace' and 'best_scheme_trace'. "
f"Got len(best_trace)={len(searcher.best_trace)}, "
f"len(best_scheme_trace)={len(searcher.best_scheme_trace)}. "
"These traces must be kept in sync by the searcher implementation."
)
constr_eval = eval_map[constraint_name]
for idx, best_acc in enumerate(searcher.best_trace):
scheme = searcher.best_scheme_trace[idx]
constr_val = constr_eval(scheme) if scheme is not None else None
history.append({
"iter": idx + 1,
"accuracy": float(best_acc) if best_acc is not None else None,
"constraint": float(constr_val) if constr_val is not None else None,
"scheme": scheme
})
return history
```
</issue_to_address>
### Comment 2
<location path="examples/mpq_dashboard.py" line_range="9-16" />
<code_context>
+ parser = argparse.ArgumentParser(description="Simple dashboard for MPQ search histories.")
+ parser.add_argument("input_json", type=str, help="Dashboard JSON file (from mpq_search/deploy scripts).")
+ parser.add_argument("--output", type=str, default="output/figs/mpq_dashboard_acc_vs_constraint.png")
+ parser.add_argument("--objective", type=str, default="accuracy")
+ parser.add_argument("--constraint", type=str, default="constraint")
+ parser.add_argument("--topk", type=int, default=10)
+ args = parser.parse_args()
+
+ runs = MiCoDashboard.load_runs(args.input_json)
+ MiCoDashboard.plot_acc_vs_constr(runs, args.objective, args.constraint, args.output)
+ MiCoDashboard.print_top_configs(runs, args.topk)
</code_context>
<issue_to_address>
**suggestion:** Consider coupling the CLI `--objective/--constraint` names with the run metadata fields.
Currently these flags only affect axis labels in `plot_acc_vs_constr`; the data always comes from the fixed `"accuracy"` and `"constraint"` history keys. This can mislead users who pass a different metric name expecting it to change which values are plotted/sorted. To make this clearer, either derive the labels from run metadata (e.g. `run["objective"]` / `run["constraint_name"]`), or let `--objective`/`--constraint` choose which history fields are used for x/y and validate those against the history structure.
Suggested implementation:
```python
args = parser.parse_args()
runs = MiCoDashboard.load_runs(args.input_json)
# Validate that the requested objective/constraint metrics exist in the run history
if not runs:
parser.error("No runs loaded from input JSON; cannot build dashboard.")
sample_run = runs[0]
history = sample_run.get("history", {})
if not isinstance(history, dict):
parser.error(
"Run history has unexpected format; expected a dict of metric_name -> values.\n"
f"Got type={type(history)} for history in first run."
)
available_metrics = set(history.keys())
if args.objective not in available_metrics:
parser.error(
f'Unknown objective metric "{args.objective}". '
f"Available metrics in history: {sorted(available_metrics)}"
)
if args.constraint not in available_metrics:
parser.error(
f'Unknown constraint metric "{args.constraint}". '
f"Available metrics in history: {sorted(available_metrics)}"
)
MiCoDashboard.plot_acc_vs_constr(runs, args.objective, args.constraint, args.output)
MiCoDashboard.print_top_configs(runs, args.topk)
```
`).
Here are the concrete edits:
<file_operations>
<file_operation operation="edit" file_path="examples/mpq_dashboard.py">
<<<<<<< SEARCH
args = parser.parse_args()
runs = MiCoDashboard.load_runs(args.input_json)
MiCoDashboard.plot_acc_vs_constr(runs, args.objective, args.constraint, args.output)
MiCoDashboard.print_top_configs(runs, args.topk)
=======
args = parser.parse_args()
runs = MiCoDashboard.load_runs(args.input_json)
# Validate that the requested objective/constraint metrics exist in the run history
if not runs:
parser.error("No runs loaded from input JSON; cannot build dashboard.")
sample_run = runs[0]
history = sample_run.get("history", {})
if not isinstance(history, dict):
parser.error(
"Run history has unexpected format; expected a dict of metric_name -> values.\n"
f"Got type={type(history)} for history in first run."
)
available_metrics = set(history.keys())
if args.objective not in available_metrics:
parser.error(
f'Unknown objective metric "{args.objective}". '
f"Available metrics in history: {sorted(available_metrics)}"
)
if args.constraint not in available_metrics:
parser.error(
f'Unknown constraint metric "{args.constraint}". '
f"Available metrics in history: {sorted(available_metrics)}"
)
MiCoDashboard.plot_acc_vs_constr(runs, args.objective, args.constraint, args.output)
MiCoDashboard.print_top_configs(runs, args.topk)
>>>>>>> REPLACE
</file_operation>
</file_operations>
<additional_changes>
To fully implement the behavior described in your review comment, `MiCoDashboard.plot_acc_vs_constr` (and any related logic) should be updated so that:
1. It uses the `objective` and `constraint` arguments as *history field names* (e.g. `history[objective]`, `history[constraint]`) rather than always reading from hard-coded `"accuracy"` / `"constraint"` keys.
2. Axis labels in the plot are derived from these names (or from more descriptive metadata fields if available), so that the plot and CLI options are consistent.
3. Any sorting/selection in `print_top_configs` that currently assumes `"accuracy"` / `"constraint"` keys should similarly be parameterized or documented, potentially by passing `objective` and `constraint` through to that function as well.
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
|
@copilot address sourcery review |
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/9ba668a1-2edb-48d6-8fa0-cc39d3cb5697 Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Addressed Sourcery review updates in |
|
@copilot I want something more real-time I guess. Something that can be displayed during the search (like scatters, curves that updates per iterations)... |
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Agent-Logs-Url: https://github.com/HKUSTGZ-MICS-LYU/MiCo-python/sessions/fd120eaa-2ed8-4116-a69f-d7a06a75505e Co-authored-by: Jzjerry <20167827+Jzjerry@users.noreply.github.com>
Implemented this in |

This PR introduces a reusable
MiCoDashboardutility for MPQ/deploy history export and visualization, standardizes searcher best-trace tracking, and adds opt-in real-time plot updates during search.What’s included
MiCoDashboardto:best_traceandbest_scheme_tracealigned via shared recording logic.examples/mpq_search.py:--live-plot--live-plot-everyValidation
py_compile) on touched files.