Skip to content

All provider #48

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
anantshri wants to merge 3 commits into
base: main
Choose a base branch
from
Open

All provider #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fe3e393
cli flag for all
anantshri Aug 8, 2026
701a511
cli flag
anantshri Aug 8, 2026
0b2c446
fix for compilation
anantshri Aug 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/sbom.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: sbom

# Reference CI caller for the aidc SBOM + license automation. All the real
# logic lives in scripts/ci/, so this workflow is a thin wrapper — the same
# scripts run unchanged under Jenkins, GitLab CI, or any other runner. See
# docs/security.md for the env-in / exit-code-out contract.

on:
push:
branches: [main]
pull_request:
branches: [main]

permissions:
contents: read

jobs:
sbom:
runs-on: ubuntu-latest
env:
# Fail the build on a license conflict in CI (the dev loop only warns).
AIDC_LICENSE_MODE: fail
# Point at a built image to also produce a build-time SBOM + code-vs-build
# diff. Leave unset to generate the code-level SBOM only.
# AIDC_IMAGE_REF: myapp:${{ github.sha }}
steps:
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6

- name: Install syft + grype (pinned + checksum-verified)
run: |
set -euo pipefail
# Direct release artifacts, verified against pinned checksums —
# no unpinned installer scripts. To bump: pick the new version from
# the tool's GitHub releases page, take the linux_amd64 sha256 from
# its checksums.txt asset, and update the four values below.
SYFT_VERSION=v1.18.1
SYFT_SHA256=066c251652221e4d44fcc4d115ce3df33a91769da38c830a8533199db2f65aab
GRYPE_VERSION=v0.87.0
GRYPE_SHA256=be710d15f5477e5c77ce03d14e480263415d7ab135e04b8483663f688823087d
mkdir -p "$HOME/.local/bin"
curl -fsSL "https://github.com/anchore/syft/releases/download/${SYFT_VERSION}/syft_${SYFT_VERSION#v}_linux_amd64.tar.gz" -o /tmp/syft.tar.gz
echo "$SYFT_SHA256 /tmp/syft.tar.gz" | sha256sum -c -
tar -xzf /tmp/syft.tar.gz -C "$HOME/.local/bin" syft
curl -fsSL "https://github.com/anchore/grype/releases/download/${GRYPE_VERSION}/grype_${GRYPE_VERSION#v}_linux_amd64.tar.gz" -o /tmp/grype.tar.gz
echo "$GRYPE_SHA256 /tmp/grype.tar.gz" | sha256sum -c -
tar -xzf /tmp/grype.tar.gz -C "$HOME/.local/bin" grype
echo "$HOME/.local/bin" >> "$GITHUB_PATH"

# - name: Build image (uncomment to enable build-time SBOM + diff)
# run: docker build -t "$AIDC_IMAGE_REF" .

- name: Generate SBOMs + license check
run: ./scripts/ci/aidc-sbom-all.sh

- name: Upload SBOM artifacts
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: sbom
path: sbom/
40 changes: 40 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Changelog

All notable changes to this project are documented here.
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Keep this file high-level: one bullet per user-visible change, grouped under the
right heading. Record the blow-by-blow detail (commands, diffs, reasoning) in
`DETAILED_CHANGELOG.md` instead.

## [Unreleased]

### Added

- `--validate all` keyword that expands to every supported provider (all 19),
instead of only the six core providers scanned by default.
- `--csv FILE` option that appends a one-line CSV summary per key: key path,
SHA256 fingerprint, then a `provider:username` entry for each identified
account, or a single `N` when no username is found.

### Changed

- `--validate` now takes a single comma-separated value (e.g. `github,gitlab`)
instead of space-separated tokens. This lets the key file follow the flag —
`keychecker --validate all <file>` now works in any argument order, where
previously the flag greedily consumed the path and errored.

### Deprecated

### Removed

### Fixed

- Editable/wheel build no longer fails with "Multiple top-level packages
discovered in a flat-layout" — package discovery is now pinned to
`keychecker*` so the `logs/` directory is not treated as a package. Also
pointed `license` at the actual `LICENSE` file (was the nonexistent
`LICENSE.md`) using the SPDX-string form.

### Security
184 changes: 184 additions & 0 deletions DETAILED_CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
# Detailed Changelog

The long-form companion to `CHANGELOG.md`. Where `CHANGELOG.md` says *what*
changed in one line, this file records *why* and *how* — enough for a future
reader to audit, reproduce, or roll back any change without re-deriving it.

Add a new entry (newest first) for every meaningful change. Use the template
below; drop sections that genuinely don't apply.

---

## 2026-08-08 — Fix CI build: flat-layout package discovery and license file

**Summary:** The `lint` job failed at `uv sync --all-extras` while building the
editable install of `keychecker`. setuptools aborted with *"Multiple top-level
packages discovered in a flat-layout: ['logs', 'keychecker']"*, and separately
warned that `LICENSE.md` could not be found and that `project.license` as a TOML
table is deprecated.

**Why:** The session-log convention added a top-level `logs/` directory. With no
explicit package configuration, setuptools' automatic flat-layout discovery saw
both `keychecker/` and `logs/` as candidate top-level packages and refused to
guess. The `license = {file = "LICENSE.md"}` entry pointed at a file that does
not exist (the repo ships `LICENSE`), and the table form is deprecated in favor
of an SPDX string (setuptools>=77).

**How:** (`pyproject.toml`)
- Added `[tool.setuptools.packages.find]` with `include = ["keychecker*"]` so
only the real package is discovered; `logs/`, `tests/`, `examples/`, etc. are
ignored.
- Replaced `license = {file = "LICENSE.md"}` with the SPDX string
`license = "GPL-3.0-or-later"` (matches the GPLv3 statement in `Readme.md`)
plus `license-files = ["LICENSE"]`.
- Bumped the build requirement to `setuptools>=77` (required for the SPDX-string
`license` form).

**Commands / verification:**
- Reproduced and verified the fix in a clean venv (repo `.venv` and system uv
were stale/older than the pinned `uv>=0.12`):
`python -m venv /tmp/bv && /tmp/bv/bin/pip install "setuptools>=77" wheel build`
then `/tmp/bv/bin/python -m build --wheel -n` → `Successfully built
keychecker-1.1.0-py3-none-any.whl`.
- Inspected the wheel: `top_level.txt` contains only `keychecker`; METADATA has
`License-Expression: GPL-3.0-or-later` and `License-File: LICENSE`.
- `aidc-scan` → clean.

**Notes:** No runtime code changed, so no new tests/coverage apply. GPLv3 is
declared as `-or-later` to match the conventional GPLv3 boilerplate; switch to
`GPL-3.0-only` if the project intends to pin to exactly v3.

---

## 2026-08-08 — Make `--validate all <file>` work in any argument order

**Summary:** `keychecker --validate all ./test_keys/github_anantshri` failed
with `invalid choice: './test_keys/github_anantshri'`. Reworked `--validate`
to accept a single comma-separated value so it no longer swallows the trailing
`key_file` positional.

**Why:** `--validate` was defined with `nargs="*"` plus `choices=`, so argparse
greedily consumed every following token — including the key-file path — as a
provider name. With a trailing optional positional this is unavoidable while
`nargs="*"` is used, and it forced users to remember to put the key file
*before* `--validate`, which is not the natural order.

**How:**
- `keychecker/cli.py`: replaced `nargs="*" / choices=` on `--validate` with a
custom `type=_parse_validate` that splits one comma-separated token (e.g.
`github,gitlab`, or `all`), strips/validates each provider against
`ALL_PROVIDERS + [ALL_KEYWORD]`, and raises `argparse.ArgumentTypeError`
with a helpful message on an unknown provider. `args.validate` stays a
`list[str] | None`, so all downstream logic (`all` expansion, discovery
guard) is unchanged.
- Updated `--validate` help text, epilog examples, and the Readme usage
(space-separated → comma-separated; added the `--validate all <file>`
example).
- `tests/test_cli.py`: added regression tests for `--validate all <file>`
ordering, comma-separated multi-provider parsing, and invalid-provider
rejection; updated the discovery-guard helper to comma-join providers.

**Commands:**

```
PYTHONPATH=/workspace python -m pytest tests/test_cli.py -q # 13 passed
PYTHONPATH=/workspace python -m keychecker --validate all ./test_keys/github_anantshri --no-progress
```

**Verification:** The previously-failing command now parses (`key_file` set,
`validate=['all']`) and runs end-to-end, scanning all 19 providers and
identifying `github: anantshri`. Invalid providers (`github,bogus`) are
rejected with a clear message listing valid choices.

**Notes:** This is a small breaking change to the CLI surface — multiple
providers must now be comma-separated (`--validate github,gitlab`) rather than
space-separated (`--validate github gitlab`). This is what makes the natural
argument order unambiguous.

---

## 2026-08-08 — Add `--validate all` and `--csv` summary output

**Summary:** Added a `--validate all` keyword that scans every supported
provider (all 19), and a `--csv FILE` option that appends a machine-readable
summary row per key so results can be collected across many keys.

**Why:** The tool validated only six core providers by default and offered no
way to scan every provider in one run; users had to enumerate them by hand.
There was also no structured output for batching — only human-readable text.
Requested: a CSV where each key id maps to `N` (no username found) or one or
more `provider:username` matches.

**What changed:**
- `keychecker/cli.py`
- New module constants `ALL_PROVIDERS` (19), `DEFAULT_PROVIDERS` (6),
`ALL_KEYWORD = "all"`, replacing the inline provider lists so the parser
choices and the runtime default share one source of truth.
- `--validate` now accepts `all`; in `run_analysis` an `all` in the list
expands to `ALL_PROVIDERS`.
- New `--csv FILE` argument and `_append_csv_row()` helper (uses the stdlib
`csv` module so commas/quotes are escaped). After validation, a row is
appended: `[key_path, sha256_fingerprint, *matches]` where matches are
`provider:username` or a single `N`.
- Repository discovery guard tightened to reject `--validate all` (discovery
needs exactly one concrete provider).
- `keychecker/utils/output.py`
- New `OutputFormatter.build_csv_row()` that builds the row, counting a
provider only when it is reachable, authenticated, and resolved a username.
- `tests/test_csv_output.py`, `tests/test_cli.py` — new unit tests for the row
builder, CSV append/escaping, `all` parsing/expansion, and the discovery
guard.

**How / commands run:**
```
# functional check (network available in sandbox)
python -m keychecker test_keys/github_anantshri --validate all --csv /tmp/out.csv
# -> row: test_keys/github_anantshri,SHA256:M7vOmp...,github:anantshri
python -m keychecker test_keys/github_anantshri --no-validate --csv /tmp/out.csv
# -> appended row: ...,N

python -m pytest tests/ -q # 22 passed
python -m flake8 keychecker/ tests/ # clean
python -m black --check keychecker/ tests/
aidc-scan # semgrep/gitleaks/shellcheck/bandit clean
```

**Errors encountered & resolution:** The checked-in `.venv` was built for
macOS and is unusable on the Linux container; `uv` is pinned to a version not
installed here and `pip` is guarded. Worked around by building a throwaway
venv (`python -m venv /tmp/kcvenv`) with `cryptography`+`pytest` for
verification only — no repo files changed. `black` reformatted the discovery
guard condition; re-ran tests and flake8 after formatting.

**Verification:** 22 tests pass (16 new); flake8 + black clean; `aidc-scan`
reports no findings; end-to-end runs produced the expected match, multi-key
append, and `N` rows.

**Notes / follow-ups:** The `all`-expansion and CSV-write branches live inside
the async `run_analysis` I/O path, which has no unit harness; they are covered
by end-to-end runs plus a mirror test of the expansion logic. CSV key id uses
both the file path and the SHA256 fingerprint (two leading columns) per the
requested format.

## YYYY-MM-DD — <short title>

**Summary:** One or two sentences on what changed and the user-facing effect.

**Why:** The problem, request, or constraint that prompted this. Link the issue
/ ticket / discussion if there is one.

**What changed:**
- File-by-file or component-by-component list of the edits.

**How / commands run:**
```
# exact commands executed, with the relevant output
```

**Errors encountered & resolution:** Anything that went wrong and how it was
fixed (or why it was left as-is).

**Verification:** How the change was proven to work — tests run, scanners,
manual checks, screenshots.

**Notes / follow-ups:** Design choices, trade-offs, and anything deferred.
42 changes: 37 additions & 5 deletions Readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ _A fast CLI tool to fingerprint SSH private keys and identify which Git hosting

### 📊 Output Modes
- **Human-readable tables**: Clean, formatted output by default
- **CSV summary**: `--csv FILE` appends one machine-readable row per key
- **Exit codes**: Automation-friendly return codes
- **Verbose logging**: Debug and trace information
- **Public key export**: Save derived public keys to files
Expand Down Expand Up @@ -64,16 +65,35 @@ cd keychecker
# Analyze a private key and Validate against servers (default behavior)
keychecker ~/.ssh/id_ed25519

# Validate against specific servers only
keychecker ~/.ssh/id_ed25519 --validate github gitlab bitbucket codeberg gitea huggingface
# Validate against specific servers only (comma-separated, no spaces)
keychecker ~/.ssh/id_ed25519 --validate github,gitlab,bitbucket,codeberg,gitea,huggingface

# Validate against specific servers only
keychecker ~/.ssh/id_rsa --validate github gitlab huggingface
keychecker ~/.ssh/id_rsa --validate github,gitlab,huggingface

# Validate against every supported provider (all 19) — order doesn't matter
keychecker ~/.ssh/id_ed25519 --validate all
keychecker --validate all ~/.ssh/id_ed25519

# Append a CSV summary row (key path, fingerprint, provider:username / N)
keychecker ~/.ssh/id_ed25519 --validate all --csv results.csv

# Skip server validation (local analysis only)
keychecker ~/.ssh/id_ed25519 --no-validate
```

The `--csv` file grows by one row per run, so it composes with a shell loop to
scan a directory of keys into a single sheet:

```bash
for key in ~/keys/*; do
keychecker "$key" --validate all --csv results.csv --no-banner --no-progress
done
# results.csv:
# ~/keys/id_ed25519,SHA256:M7vOmp...,github:anantshri,gitlab:anant
# ~/keys/id_rsa,SHA256:DE8Kf...,N
```

### Repository Discovery

```bash
Expand Down Expand Up @@ -106,16 +126,25 @@ Positional Arguments:
Options:
-i, --input PATH Path to private key file (alternative to positional)

--validate SERVERS One or more servers to validate against
Choices: github, gitlab, bitbucket, codeberg, gitea, huggingface
--validate PROVIDERS Comma-separated server(s) to validate against, e.g.
"github" or "github,gitlab" (default: the six core
providers). Use "all" for every provider.
Choices: github, gitlab, bitbucket, codeberg, gitea,
huggingface, dataops, assembla, boltic, sourcehut,
notabug, azuredevops, framagit, gitverse, launchpad,
gitee, coding, codeup, gitflic, all
--no-validate Skip server validation (local analysis only)

--discovery FILE Enable repository discovery with wordlist file
(requires exactly one concrete --validate server)

--github-token TOKEN GitHub API token for enhanced organization discovery
--no-progress Disable progress bars during repository discovery

--public-out FILE Save derived public key to file
--csv FILE Append a CSV summary row for the key: key path,
SHA256 fingerprint, then one "provider:username" per
identified account, or a single "N" if none found
--no-banner Suppress banner output

--timeout SECONDS Per-connection timeout (default: 5)
Expand All @@ -138,6 +167,9 @@ keychecker ~/.ssh/id_rsa --validate github
# Validate against Hugging Face only
keychecker ~/.ssh/id_rsa --validate huggingface

# Validate against every supported provider and record a CSV row
keychecker ~/.ssh/id_rsa --validate all --csv results.csv

# Discover repositories with custom wordlist
keychecker ~/.ssh/id_rsa --validate github --discovery my_repos.txt

Expand Down
Loading
Loading