Quick one. In sdk/nexent/core/nlp/tokenizer.py:59:
tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}total_weight is only incremented inside the if word not in analyse.default_tfidf.stop_words and word.strip() branch (lines 46-56). If every token is a stop word (e.g., a query consisting only of punctuation, whitespace, or common particles like "的 是 在"), total_weight == 0.0 and term_stats is empty.
In that case the dict comprehension is empty (no division), so today the function silently returns {} from the meaningful-terms guard at line 98 — but the moment any non-stop token slips in, the division will execute, and any future change that pre-populates term_stats without updating total_weight will raise ZeroDivisionError. The dependency between the two accumulators is fragile and undocumented.
Suggested fix: guard explicitly.
if total_weight == 0.0:
return {}
tf_weights = {term: weight / total_weight for term, weight in term_stats.items()}This makes the contract obvious and prevents the latent footgun.
Quick one. In
sdk/nexent/core/nlp/tokenizer.py:59:total_weightis only incremented inside theif word not in analyse.default_tfidf.stop_words and word.strip()branch (lines 46-56). If every token is a stop word (e.g., a query consisting only of punctuation, whitespace, or common particles like"的 是 在"),total_weight == 0.0andterm_statsis empty.In that case the dict comprehension is empty (no division), so today the function silently returns
{}from the meaningful-terms guard at line 98 — but the moment any non-stop token slips in, the division will execute, and any future change that pre-populatesterm_statswithout updatingtotal_weightwill raiseZeroDivisionError. The dependency between the two accumulators is fragile and undocumented.Suggested fix: guard explicitly.
This makes the contract obvious and prevents the latent footgun.