Skip to content

scoring: switch canonical sentence model to bge-small-en-v1.5 @ 0.72 #209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
that-guy-wade merged 1 commit into from
Jun 30, 2026
Merged

scoring: switch canonical sentence model to bge-small-en-v1.5 @ 0.72 #209

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions docker/validator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,9 @@ WORKDIR /app
COPY docker/validator/pyproject.toml docker/validator/uv.lock /app/
RUN uv sync --frozen --no-dev --no-install-project

# Pre-cache SentenceTransformer models (avoids runtime download). The
# canonical Qwen3 model is always loaded; the bge-small bundle is for the
# optional ORO-1474 shadow scorer enabled via SHADOW_SCORE_MODEL at runtime.
# Pre-cache the canonical sentence model (avoids runtime download).
ARG HF_TOKEN=""
RUN HF_HUB_ENABLE_HF_TRANSFER=0 HF_TOKEN=${HF_TOKEN} .venv/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('Qwen/Qwen3-Embedding-0.6B'); SentenceTransformer('BAAI/bge-small-en-v1.5')"
RUN HF_HUB_ENABLE_HF_TRANSFER=0 HF_TOKEN=${HF_TOKEN} .venv/bin/python -c "from sentence_transformers import SentenceTransformer; SentenceTransformer('BAAI/bge-small-en-v1.5')"

# Copy validator code
COPY subnet/ /app/subnet/
Expand Down
11 changes: 4 additions & 7 deletions src/agent/rewards/orm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
import os
from collections import Counter

SENTENCE_MODEL_NAME = "Qwen/Qwen3-Embedding-0.6B"
TITLE_SIM_THRESHOLD = 0.7
SENTENCE_MODEL_NAME = "BAAI/bge-small-en-v1.5"
TITLE_SIM_THRESHOLD = 0.72

# ORO-1474 shadow scoring: when SHADOW_SCORE_MODEL is set, encode the same
# Shadow scoring: when SHADOW_SCORE_MODEL is set, encode the same
# (product_title, gt_title) pair with the candidate model and log the
# canonical + shadow cosine similarity so we can grep CW Logs Insights and
# replay any threshold offline before promoting a model swap. Threshold is
Expand All @@ -33,10 +33,7 @@ def _get_sentence_model():
try:
from sentence_transformers import SentenceTransformer

_sentence_model = SentenceTransformer(
SENTENCE_MODEL_NAME,
model_kwargs={"torch_dtype": "bfloat16"},
)
_sentence_model = SentenceTransformer(SENTENCE_MODEL_NAME)
except ImportError:
_sentence_model_unavailable = True
return None
Expand Down
2 changes: 1 addition & 1 deletion tests/test_scoring_perf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ def test_non_gt_calls_encode(mock_get):
)

assert m.encode.call_count >= 1
assert hits["title"] == 1 # similarity 0.95 >= TITLE_SIM_THRESHOLD (0.7)
assert hits["title"] == 1 # similarity 0.95 >= TITLE_SIM_THRESHOLD (0.72)
assert hits["price"] == 1 # 15 <= 20
assert hits["service"] == 1
assert score == 1.0 # 3/3
Expand Down
Loading