Skip to content

Commit 4aea6b4

Browse files
h3xxitclaude
andcommitted
docs: the spec extractor renders what the docstrings say
Four defects in scripts/extract_required_docs.py, each surfacing as a wrong page in the generated spec (cubic, utcp-specification #66): - A prose line containing a colon that is not a 'name: description' pair was silently DROPPED: '(note: ...)' sentences, quoted URLs after '(e.g.', anything with a colon mid-sentence. Such a line is now ordinary text. - Any line ending in a colon became a title-cased section header, code spans included ('Inheritance is controlled by `inherit_env_vars`:' -> '**Inheritance Is Controlled By `Inherit_Env_Vars`**'; 'def tool1():' inside an example -> '**Def Tool1()**'). A header is now a known Google-style one or a short title of words, digits, spaces and hyphens. - A docstring with no section header at all was never flushed, so 62 REQUIRED docstrings across the spec rendered as '*No ... documentation available*' (UtcpClient, every auth serializer, the plugin loader, all socket methods, ...). The bogus headers above had been flushing some of them by accident, which the fix exposed. - Cross-reference links were inserted inside inline code spans, where Markdown shows them as literal brackets. Links now stop at code spans; the field-list placeholder backticks are unwrapped before the pass so fields keep their links. Also: a class whose own docstring is not REQUIRED but whose methods are now renders those methods (the index already counted them), and index links are POSIX paths on every platform. Two docstrings corrected on the way: the CLI template claimed tool_args are 'shell-quoted' - the mechanism is per-invocation environment variables, as the same docstring explains - and 'OAuth2Auth.cache_key' is now written so the class links and cache_key stays code. Regenerated against the published pages: 43 pages change; every removed line is a placeholder, a bogus header, a link-in-code-span or a mangled example fragment; 360 lines of previously invisible documentation come back. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 6b4f364 commit 4aea6b4

4 files changed

Lines changed: 123 additions & 83 deletions

File tree

‎plugins/communication_protocols/cli/src/utcp_cli/cli_call_template.py‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -147,8 +147,10 @@ class CliCallTemplate(CallTemplate):
147147
commands: A list of CommandStep objects defining the commands to execute
148148
in order. Each command can contain UTCP_ARG_argname_UTCP_END placeholders
149149
that will be replaced with values from tool_args during execution.
150-
Placeholders are shell-quoted and therefore expand to exactly one
151-
shell token (see class docstring).
150+
Each placeholder becomes a shell-variable reference whose value
151+
reaches the subprocess through a per-invocation environment
152+
variable, so it expands to exactly one shell token and cannot be
153+
reinterpreted as shell syntax (see class docstring).
152154
env_vars: A dictionary of environment variables to set for the command's
153155
execution context. Values can be static strings or placeholders for
154156
variables from the UTCP client's variable substitutor. Always
@@ -238,9 +240,11 @@ class CliCallTemplate(CallTemplate):
238240
Security Considerations:
239241
- Commands are executed in a subprocess. Ensure that the commands
240242
specified are from a trusted source.
241-
- `tool_args` values are shell-quoted on substitution, but the
242-
*command template itself* is not — never assemble it from
243-
untrusted input.
243+
- `tool_args` values never touch the command text: they reach the
244+
subprocess through per-invocation environment variables and the
245+
shell expands them only after it has parsed the script. The
246+
*command template itself* has no such protection — never assemble
247+
it from untrusted input.
244248
- The host environment is restricted; secrets are not propagated
245249
unless explicitly named in `env_vars` or `inherit_env_vars`.
246250
- Commands should use the appropriate syntax for the target platform

‎plugins/communication_protocols/http/src/utcp_http/http_communication_protocol.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ class HttpCommunicationProtocol(CommunicationProtocol):
6363
6464
Attributes:
6565
_session: Optional aiohttp ClientSession for connection reuse.
66-
_oauth_tokens: Cache of OAuth2 tokens keyed by the full credential configuration (``OAuth2Auth.cache_key``).
66+
_oauth_tokens: Cache of OAuth2 tokens keyed by the full credential configuration (OAuth2Auth's ``cache_key``).
6767
_log: Logger function for debugging and error reporting.
6868
"""
6969

‎plugins/communication_protocols/websocket/src/utcp_websocket/websocket_communication_protocol.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ class WebSocketCommunicationProtocol(CommunicationProtocol):
6363
Attributes:
6464
_connections: Active WebSocket connections by provider key.
6565
_sessions: aiohttp ClientSessions for connection management.
66-
_oauth_tokens: Cache of OAuth2 tokens keyed by the full credential configuration (``OAuth2Auth.cache_key``).
66+
_oauth_tokens: Cache of OAuth2 tokens keyed by the full credential configuration (OAuth2Auth's ``cache_key``).
6767
"""
6868

6969
def __init__(self, logger_func: Optional[Callable[[str], None]] = None):

‎scripts/extract_required_docs.py‎

Lines changed: 112 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -148,44 +148,49 @@ def process_section_content(content_lines):
148148
line = line.replace('{', '\\{').replace('}', '\\}')
149149
stripped = line.strip()
150150

151-
# Check if this looks like a parameter/item definition (name: description)
151+
# Check if this looks like a parameter/item definition (name: description).
152+
# A line with a colon that is NOT one -- prose such as "(note: ...)" or a
153+
# quoted URL -- is ordinary text and falls through to the branches below;
154+
# it must never be dropped.
155+
param_match = None
152156
if ':' in stripped and not stripped.endswith(':'):
153157
colon_pos = stripped.find(':')
154-
param_name = stripped[:colon_pos].strip()
155-
param_desc = stripped[colon_pos + 1:].strip()
156-
157-
# Check if param_name looks like a parameter (no spaces, reasonable length)
158-
if ' ' not in param_name and len(param_name) <= 50 and param_name.replace('_', '').isalnum():
159-
# This is likely a parameter definition
160-
processed.append(f"- **`{param_name}`**: {param_desc}")
161-
162-
# Check for continuation lines (indented more than the parameter line)
163-
base_indent = len(line) - len(line.lstrip())
164-
i += 1
165-
while i < len(content_lines):
166-
next_line = content_lines[i]
167-
next_stripped = next_line.strip()
168-
next_indent = len(next_line) - len(next_line.lstrip()) if next_stripped else 0
169-
170-
# Check if we hit a code block
171-
if next_stripped.startswith('```'):
172-
break
173-
174-
if not next_stripped:
175-
# Empty line - add it and continue
176-
processed.append('')
177-
i += 1
178-
elif next_indent > base_indent:
179-
# Continuation line - add with proper spacing
180-
processed.append(f" {next_stripped}")
181-
i += 1
182-
else:
183-
# Not a continuation, back up and break
184-
break
185-
continue
186-
158+
candidate = stripped[:colon_pos].strip()
159+
# A parameter name has no spaces and is a plain identifier
160+
if ' ' not in candidate and len(candidate) <= 50 and candidate.replace('_', '').isalnum():
161+
param_match = (candidate, stripped[colon_pos + 1:].strip())
162+
163+
if param_match is not None:
164+
param_name, param_desc = param_match
165+
processed.append(f"- **`{param_name}`**: {param_desc}")
166+
167+
# Check for continuation lines (indented more than the parameter line)
168+
base_indent = len(line) - len(line.lstrip())
169+
i += 1
170+
while i < len(content_lines):
171+
next_line = content_lines[i]
172+
next_stripped = next_line.strip()
173+
next_indent = len(next_line) - len(next_line.lstrip()) if next_stripped else 0
174+
175+
# Check if we hit a code block
176+
if next_stripped.startswith('```'):
177+
break
178+
179+
if not next_stripped:
180+
# Empty line - add it and continue
181+
processed.append('')
182+
i += 1
183+
elif next_indent > base_indent:
184+
# Continuation line - add with proper spacing
185+
processed.append(f" {next_stripped}")
186+
i += 1
187+
else:
188+
# Not a continuation, back up and break
189+
break
190+
continue
191+
187192
# Check if line starts with a list marker
188-
elif stripped.startswith(('- ', '* ', '+ ')):
193+
if stripped.startswith(('- ', '* ', '+ ')):
189194
# This is already a markdown list item
190195
processed.append(stripped)
191196
elif stripped.startswith(('1. ', '2. ', '3. ', '4. ', '5. ', '6. ', '7. ', '8. ', '9. ')):
@@ -204,9 +209,13 @@ def process_section_content(content_lines):
204209
# Parse the docstring line by line
205210
for line in lines:
206211
stripped_lower = line.strip().lower()
207-
208-
# Check if this line is a section header
209-
if stripped_lower in section_headers or stripped_lower.endswith(':'):
212+
213+
# Check if this line is a section header: a known Google-style header, or a
214+
# short title made of words only ("Security Considerations:"). A sentence
215+
# that merely ends in a colon ("Inheritance is controlled by `x`:") is
216+
# content -- treating it as a header would title-case it, code span included.
217+
is_custom_header = re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9 \-]{0,40}:', line.strip()) is not None
218+
if stripped_lower in section_headers or is_custom_header:
210219
# Save previous section if it exists
211220
if current_section:
212221
processed_content = process_section_content(current_section_content)
@@ -231,6 +240,10 @@ def process_section_content(content_lines):
231240
if processed_content:
232241
result.append(f"\n**{current_section.title()}**\n")
233242
result.extend(processed_content)
243+
else:
244+
# No section header anywhere: the whole docstring is the preamble,
245+
# which is otherwise only flushed when a header follows it.
246+
result.extend(process_section_content(current_section_content))
234247

235248
# Clean up the result
236249
final_result = []
@@ -530,18 +543,46 @@ def add_cross_references_post_generation(self, text: str, current_output_file: s
530543
class_anchor = re.sub(r'[^\w\-_]', '-', class_name.lower()).strip('-')
531544
link = f"[{class_name}](./{relative_path_str}#{class_anchor})"
532545

533-
# Don't replace matches that are in code blocks
546+
# Don't replace matches that are in code blocks or inline code spans
534547
lines = modified_text.split('\n')
535548
in_code_block = False
536549
for i, line in enumerate(lines):
537550
if line.strip().startswith('```'):
538551
in_code_block = not in_code_block
539552
elif not in_code_block:
540-
lines[i] = re.sub(pattern, link, line)
553+
lines[i] = self._sub_outside_inline_code(pattern, link, line)
541554
modified_text = '\n'.join(lines)
542-
555+
543556
return modified_text
557+
558+
@staticmethod
559+
def _sub_outside_inline_code(pattern: str, replacement: str, line: str) -> str:
560+
"""Substitute only outside `...` / ``...`` spans.
561+
562+
Markdown renders a code span literally, so a link inserted inside one shows up
563+
as raw brackets instead of a link.
564+
"""
565+
parts = re.split(r'(`+[^`]*`+)', line)
566+
return ''.join(part if part.startswith('`') else re.sub(pattern, replacement, part) for part in parts)
544567

568+
def _render_methods(self, content: List[str], methods: List[DocEntry], file_path: str) -> None:
569+
"""Append a class's documented methods to ``content``."""
570+
content.extend(["#### Methods:", ""])
571+
for method in methods:
572+
# Add cross-references to method signature
573+
linked_signature = self.add_cross_references(method.signature, file_path)
574+
docstrings = method.docstring if method.docstring else "*No method documentation available*"
575+
content.extend(
576+
[
577+
"<details>",
578+
f"<summary>{linked_signature}</summary>",
579+
"",
580+
docstrings,
581+
"</details>",
582+
"",
583+
]
584+
)
585+
545586
def generate_module_markdown(self, file_path: str, file_data: Dict[str, List[DocEntry]]) -> str:
546587
"""Generate markdown content for a single module/file."""
547588
if not any(file_data.values()):
@@ -632,34 +673,27 @@ def generate_module_markdown(self, file_path: str, file_data: Dict[str, List[Doc
632673

633674
# Add methods for this class
634675
if class_entry.name in methods_by_class:
635-
content.extend(["#### Methods:", ""])
636-
637-
for method in methods_by_class[class_entry.name]:
638-
method_anchor = re.sub(r'[^\w\-_]', '-', f"{class_entry.name}-{method.name}".lower()).strip('-')
639-
640-
# Add cross-references to method signature
641-
linked_signature = self.add_cross_references(method.signature, file_path)
642-
643-
docstrings = ""
644-
645-
if method.docstring:
646-
docstrings = method.docstring
647-
else:
648-
docstrings = "*No method documentation available*"
649-
650-
content.extend(
651-
[
652-
"<details>",
653-
f"<summary>{linked_signature}</summary>",
654-
"",
655-
docstrings,
656-
"</details>",
657-
"",
658-
]
659-
)
660-
676+
self._render_methods(content, methods_by_class[class_entry.name], file_path)
677+
661678
content.extend(["---", ""])
662-
679+
680+
# A class whose own docstring is not REQUIRED can still have REQUIRED
681+
# methods. They are required documentation and the index counts them,
682+
# so they are rendered under a bare class heading rather than lost.
683+
documented_classes = {class_entry.name for class_entry in file_data['classes']}
684+
for class_name, methods in methods_by_class.items():
685+
if class_name in documented_classes:
686+
continue
687+
class_anchor = re.sub(r'[^\w\-_]', '-', class_name.lower()).strip('-')
688+
content.extend([
689+
f"### class {class_name} {{#{class_anchor}}}",
690+
"",
691+
"*No class documentation available*",
692+
"",
693+
])
694+
self._render_methods(content, methods, file_path)
695+
content.extend(["---", ""])
696+
663697
# Add standalone functions
664698
if file_data['functions']:
665699
for func_entry in file_data['functions']:
@@ -748,7 +782,7 @@ def generate_index_file(self, modules: Dict[str, Dict[str, List[DocEntry]]], out
748782
index_path = output_path / "index.md"
749783
target_path = Path(output_file_path)
750784
try:
751-
relative_path = target_path.relative_to(output_path)
785+
relative_path = target_path.relative_to(output_path).as_posix()
752786
link_path = f"./{relative_path}"
753787
except ValueError:
754788
# Fallback to simple filename if relative path calculation fails
@@ -805,7 +839,7 @@ def generate_index_file(self, modules: Dict[str, Dict[str, List[DocEntry]]], out
805839
index_path = output_path / "index.md"
806840
target_path = Path(output_file_path)
807841
try:
808-
relative_path = target_path.relative_to(output_path)
842+
relative_path = target_path.relative_to(output_path).as_posix()
809843
link_path = f"./{relative_path}"
810844
except ValueError:
811845
# Fallback to simple filename if relative path calculation fails
@@ -900,10 +934,11 @@ def generate_docs(self, output_dir: str) -> None:
900934

901935
# Second pass: Add cross-references and write files
902936
for file_path, (content, output_file_path) in generated_files.items():
903-
# Post-process content to add proper cross-references
904-
processed_content = self.add_cross_references_post_generation(content, str(output_file_path).replace('\\', '/'))
905-
# Also process field references
906-
lines = processed_content.split('\n')
937+
# Field lines were emitted as "- `name: type`" -- the backticks are a
938+
# placeholder (see format_field_with_references), not a code span.
939+
# Unwrap them BEFORE cross-referencing: the cross-reference pass
940+
# leaves real code spans alone, and these must receive links.
941+
lines = content.split('\n')
907942
processed_lines = []
908943
for line in lines:
909944
if line.strip().startswith('- `') and ':' in line:
@@ -916,8 +951,9 @@ def generate_docs(self, output_dir: str) -> None:
916951
processed_lines.append(line)
917952
else:
918953
processed_lines.append(line)
919-
920-
final_content = '\n'.join(processed_lines)
954+
955+
# Post-process content to add proper cross-references
956+
final_content = self.add_cross_references_post_generation('\n'.join(processed_lines), str(output_file_path).replace('\\', '/'))
921957

922958
with open(output_file_path, 'w', encoding='utf-8') as f:
923959
f.write(final_content)

0 commit comments

Comments
 (0)