🐛 Fix term position weighting

[original4422]

Summary

• filter whitespace and stop words before calculating token positions
• use the retained token count to boost the first and last three terms
• add a deterministic ten-token regression test covering filtered entries and both position boundaries

Root cause

calculate_term_weights compared a token index with one third of the raw character length. Character length is unrelated to the retained token stream, so end-of-query terms were not weighted consistently. The loop index also included tokens later discarded as whitespace or stop words.

Validation

• uv run --no-project --with pytest --with jieba python -m pytest test/sdk/core/nlp/test_tokenizer.py -q (1 passed)
• baseline comparison for empty input, TF/POS/stop-word/length/normalization, IDF, and proper-noun enhancement
• uv run --no-project --with ruff ruff check --ignore I001 sdk/nexent/core/nlp/tokenizer.py test/sdk/core/nlp/test_tokenizer.py
• uv run --no-project --with ruff ruff format --check test/sdk/core/nlp/test_tokenizer.py
• git diff --check
• python3 -m compileall -q sdk/nexent/core/nlp/tokenizer.py test/sdk/core/nlp/test_tokenizer.py

Fixes #3811

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[original4422]

@WMC001 Hi! When you have a moment, could you please check whether this PR is ready to merge or let me know if any changes are needed? Thank you!

[original4422]

@WMC001 Hi, when you have a moment, could you please check whether this PR is ready to merge or needs any changes? Thank you!