Skip to content

Commit accc9de

Browse files
ralyodioclaude
andauthored
feat: the directory hosts webrings (OpenWebring), one ring per topic (#196)
rssamplifier.com is the first OpenWebring host (logicsrc.com/openwebring): a ring is an ordered, circular list of member sites, and every member carries its own word on who makes it, human, ai or both, read from the member's /.well-known/openwebring.json and never verified. - rings + ring_members (explicit position: a topic's feeds are re-derived every crawl, so ring order must be its own column; new members append). - /.well-known/openwebring.json (a route: rings are data), /ring index, /ring/<slug> page with the join snippet, /ring/<slug>/openwebring.json, /ring/<slug>/opml, hops /next /previous /prev /random with ?from= and every shape older rings use (?host=, ?via=, ?url=, /<member>/next, the bare Referer); unknown from is random; wrap; skip inactive; 302 no-store. - Verification in the poller (RING_* env): a bounded batch per tick, one stranger's front page at a time, any link to the ring counts, the member's descriptor read for made_by and rings[]; inactive, never removed. An on-demand /ring/<slug>/check behind the page gate. - Hops exempt from the proxy throttle (a refused hop is a broken link on someone else's site), counted like everything else. - Owner override PUT /api/rings/<slug>/members/<member>; OpenAccess scope openwebring:edit; /api/rings, /api/rings/<slug>; MCP list_rings, get_ring, ring_next; nav, sitemap, seed script. - linksBack/fetchPage moved to @rssamplifier/feed (linksTo, fetchPage) so the poller and the web app run one scan. Claude-Session: https://claude.ai/code/session_01XYae2mH3khdwiXUVzcVMDw Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
1 parent 8f9a95e commit accc9de

41 files changed

Lines changed: 3375 additions & 39 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.env.example‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,22 @@ CARD_BACKFILL=1
5252
CARD_BACKFILL_BATCH=8
5353
CARD_BACKFILL_SECONDS=20
5454

55+
# OpenWebring (logicsrc.com/openwebring). The crawler seeds one ring per
56+
# well-covered topic and reads every member's front page for a link back, one
57+
# site at a time, oldest verdict first. RING_VERIFY=0 pauses the pass and the
58+
# seeding without changing their tuning. RING_TOPICS is how many topics get a
59+
# ring (the most covered, at least five eligible feeds each); RING_BATCH
60+
# members are checked per pass, every RING_SECONDS; a verdict stands
61+
# RING_RECHECK_DAYS before the member is looked at again; the rings are
62+
# re-seeded every RING_SEED_SECONDS so a feed new to a topic joins at the end.
63+
# One-off: `node --env-file=.env scripts/seed-rings.mjs`.
64+
RING_VERIFY=1
65+
RING_TOPICS=20
66+
RING_BATCH=25
67+
RING_SECONDS=60
68+
RING_RECHECK_DAYS=7
69+
RING_SEED_SECONDS=21600
70+
5571
# Nonessential maintenance can be paused while a large first-crawl backlog owns
5672
# the database write path. The work is resumable when these are switched on.
5773
CLUSTER_BACKFILL=1

‎apps/poller/src/index.js‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,8 @@ import {
2121
drainDiscoveryQueue,
2222
drainDiscoveryKeywords,
2323
drainImport,
24+
seedTopRings,
25+
verifyRingMembers,
2426
} from '@rssamplifier/ingest';
2527
import { runDueSources, discoverFromOwnTopics } from '@rssamplifier/discover';
2628
import { findFeedCard } from '@rssamplifier/feed';
@@ -97,6 +99,21 @@ const cardEnabled = env['CARD_BACKFILL'] !== '0' && env['CARD_BACKFILL'] !== 'fa
9799
// that only happens if the process lives long enough to reach it.
98100
const cardIntervalMs = (Number(env['CARD_BACKFILL_SECONDS']) || 20) * 1000;
99101

102+
// OpenWebring (logicsrc.com/openwebring). The rings are seeded from the most
103+
// covered topics and every member's front page is read for a link back, one
104+
// site at a time, oldest verdict first; see packages/ingest/src/webring.js.
105+
// How many members a pass checks, how often a pass runs, how long a verdict
106+
// stands before the member is looked at again, how many topics get a ring,
107+
// and how often the rings are re-seeded so a feed new to a topic joins at
108+
// the end. RING_VERIFY=0 pauses all of it without changing the tuning.
109+
const ringEnabled = env['RING_VERIFY'] !== '0' && env['RING_VERIFY'] !== 'false';
110+
const ringBatch = Number(env['RING_BATCH']) || 25;
111+
const ringIntervalMs = (Number(env['RING_SECONDS']) || 60) * 1000;
112+
const ringRecheckDays = Number(env['RING_RECHECK_DAYS']) || 7;
113+
const ringTopics = Number(env['RING_TOPICS']) || 20;
114+
const ringSeedMs = (Number(env['RING_SEED_SECONDS']) || 21_600) * 1000;
115+
const ringBase = (env['SITE_URL'] || 'https://rssamplifier.com').replace(/\/+$/, '');
116+
100117
// Items keyed for grouping per tick. Each one is a title hashed in process and
101118
// a single-column update, so the batch is bounded by the write round trip
102119
// rather than by CPU.
@@ -245,6 +262,10 @@ let carding = false;
245262
// Guards the enrichment pass against overlapping itself. A batch of five feeds
246263
// that each need four fetches of a slow server can outlast its own interval.
247264
let enriching = false;
265+
// The same guard for the webring pass, which fetches strangers' front pages
266+
// one at a time and must not stack, and when the rings were last seeded.
267+
let ringing = false;
268+
let lastRingSeed = 0;
248269

249270
// And for the alert pass, where stacking would be worse than wasteful: two
250271
// overlapping passes read the same watermark and would send the same digest
@@ -756,6 +777,48 @@ async function cardTick() {
756777
}
757778
}
758779

780+
/**
781+
* Keep the webrings seeded and verified.
782+
*
783+
* Two jobs on one timer. Seeding runs every RING_SEED_SECONDS: one ring per
784+
* well-covered topic, existing members keeping their place and new feeds
785+
* appended (webrings.seedTopicRing). Verification runs every tick: a bounded
786+
* batch of the members whose verdict is oldest, each one a fetch of their
787+
* front page and of their descriptor, sequentially, because every one is a
788+
* request to a stranger's server and the crawl beside this already owns the
789+
* outbound budget. The hop routes never do any of this; they read only what
790+
* this pass last wrote, which is what keeps a hop a cheap 302.
791+
*/
792+
async function ringTick() {
793+
if (!ringEnabled || stopping || ringing) return;
794+
ringing = true;
795+
796+
try {
797+
if (Date.now() - lastRingSeed >= ringSeedMs) {
798+
lastRingSeed = Date.now();
799+
const seeded = await seedTopRings(db, { topics: ringTopics, minMembers: 5 });
800+
// Logged only when something changed: once the rings exist, a line
801+
// every six hours saying "0 added" is not a log.
802+
if (seeded.created || seeded.added) log('rings-seeded', seeded);
803+
}
804+
805+
const result = await verifyRingMembers(db, {
806+
batch: ringBatch,
807+
base: ringBase,
808+
recheckDays: ringRecheckDays,
809+
onEvent: publishLog ? recorder.record : null,
810+
stopping: () => stopping,
811+
});
812+
// Silent when nothing was due, which is the steady state between recheck
813+
// periods.
814+
if (result.checked) log('rings', result);
815+
} catch (err) {
816+
log('rings-error', { message: String(err?.message ?? err) });
817+
} finally {
818+
ringing = false;
819+
}
820+
}
821+
759822
/**
760823
* Tell people about the posts they asked to be told about.
761824
*
@@ -1047,6 +1110,7 @@ const backfillTimer = setInterval(backfillTick, clusterIntervalMs);
10471110
const cardTimer = setInterval(cardTick, cardIntervalMs);
10481111
const alertTimer = setInterval(alertTick, alertIntervalMs);
10491112
const enrichTimer = setInterval(enrichTick, authorIntervalMs);
1113+
const ringTimer = setInterval(ringTick, ringIntervalMs);
10501114
const queueTimer = setInterval(queueTick, queueSampleMs);
10511115
const statsTimer = setInterval(statsTick, statsWarmMs);
10521116
const searchTimer = setInterval(searchTick, searchIntervalMs);
@@ -1058,6 +1122,7 @@ void backfillTick();
10581122
void cardTick();
10591123
void alertTick();
10601124
void enrichTick();
1125+
void ringTick();
10611126
void queueTick();
10621127
// Run once at boot, so a deploy does not leave the page slow until the first
10631128
// interval comes round.
@@ -1071,6 +1136,8 @@ log('started', {
10711136
clusterBackfill,
10721137
cardBatch: cardEnabled ? cardBatch : 0,
10731138
authorBatch: authorEnabled ? authorBatch : 0,
1139+
ringBatch: ringEnabled ? ringBatch : 0,
1140+
ringTopics: ringEnabled ? ringTopics : 0,
10741141
crawlAutocommit: ['1', 'true'].includes(String(env['TURSO_CRAWL_AUTOCOMMIT'] ?? '').toLowerCase()),
10751142
auxiliaryWrites: !['0', 'false'].includes(String(env['CRAWL_AUXILIARY_WRITES'] ?? '').toLowerCase()),
10761143
catchupOnly,
@@ -1091,6 +1158,7 @@ function shutdown(signal) {
10911158
clearInterval(backfillTimer);
10921159
clearInterval(cardTimer);
10931160
clearInterval(enrichTimer);
1161+
clearInterval(ringTimer);
10941162
clearInterval(alertTimer);
10951163
clearInterval(queueTimer);
10961164
clearInterval(statsTimer);

‎apps/web/public/.well-known/openaccess.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,8 @@
1919
]
2020
},
2121
"scopes": {
22-
"openprofile:edit": "Correct an author's OpenProfile.md you have claimed: PUT /api/authors/{slug}/openprofile (text/markdown, the whole file, or a JSON patch), POST /api/authors/{slug}/claim. The grant's principal is recorded as the owner; a claim is verified by the address the author published or by their site linking back."
22+
"openprofile:edit": "Correct an author's OpenProfile.md you have claimed: PUT /api/authors/{slug}/openprofile (text/markdown, the whole file, or a JSON patch), POST /api/authors/{slug}/claim. The grant's principal is recorded as the owner; a claim is verified by the address the author published or by their site linking back.",
23+
"openwebring:edit": "State who makes a site of yours that is in one of this host's webrings: PUT /api/rings/{ring}/members/{slug} with a JSON body { made_by: human | ai | both | null, disclosure }. Recorded as the owner's word, which a later read of the site's own /.well-known/openwebring.json never overwrites."
2324
},
2425
"honours": [
2526
"profullstack.com/all-access"

‎apps/web/public/.well-known/openmcp.json‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,10 @@
2929
"submit_feed",
3030
"get_openprofile",
3131
"update_openprofile",
32-
"claim_openprofile"
32+
"claim_openprofile",
33+
"list_rings",
34+
"get_ring",
35+
"ring_next"
3336
],
3437
"catalogs": [
3538
"https://openmcp.logicsrc.com"
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
import { webrings } from '@rssamplifier/db';
2+
3+
import { db, siteUrl } from '../../../lib/db.js';
4+
import { hostFile } from '../../../lib/openwebring.js';
5+
6+
export const dynamic = 'force-dynamic';
7+
8+
/**
9+
* /.well-known/openwebring.json: the rings this site hosts.
10+
*
11+
* A route rather than a file in /public, because rings are data: which
12+
* topics have one and how many members each holds changes as the poller
13+
* seeds and verifies, and a file would say whatever it said on deploy day.
14+
* Five minutes of caching is the same allowance the JSON API gives itself.
15+
*/
16+
export async function GET() {
17+
const rings = await webrings.listRings(db());
18+
const body = hostFile({ base: siteUrl(), siteName: 'RSS Amplifier', rings });
19+
20+
return new Response(JSON.stringify(body, null, 2), {
21+
headers: {
22+
'content-type': 'application/json; charset=utf-8',
23+
'access-control-allow-origin': '*',
24+
'cache-control': 'public, max-age=300',
25+
},
26+
});
27+
}
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
import { webrings } from '@rssamplifier/db';
2+
3+
import { db } from '../../../../../../lib/db.js';
4+
import { forgetRing, loadRing, profilesForFeed } from '../../../../../../lib/rings.js';
5+
import { madeByPatch, memberEditVerdict, memberEntry } from '../../../../../../lib/openwebring.js';
6+
import { adminEmails, callerOf } from '../../../../../../lib/profileAuth.js';
7+
import { json } from '../../../../authors/route.js';
8+
9+
export const dynamic = 'force-dynamic';
10+
11+
/** The most a patch may be. Two fields; a page of them is somebody probing. */
12+
const BODY_LIMIT = 4 * 1024;
13+
14+
/**
15+
* One member of a ring, read and corrected.
16+
*
17+
* GET is the member's entry from the ring file.
18+
*
19+
* PUT is the owner's word on who makes the site: `{ "made_by": "human" |
20+
* "ai" | "both" | null, "disclosure": "none" | "ai-assisted" | "ai-generated"
21+
* | "autonomous" | null }`. Recorded as the owner's (or an admin's), which a
22+
* later read of the site's own descriptor never overwrites. Auth: the site's
23+
* session, an API key, or an OpenAccess bearer; the caller must have claimed
24+
* the profile of one of the feed's authors, be an admin, or hold the
25+
* `openwebring:edit` scope.
26+
*
27+
* @param {Request} req
28+
* @param {{ params: Promise<{ slug: string, member: string }> }} ctx
29+
*/
30+
export async function GET(req, { params }) {
31+
const { slug, member } = await params;
32+
const loaded = await loadRing(slug);
33+
const row = loaded?.members.find((m) => m.member_slug === member.toLowerCase());
34+
if (!loaded || !row) return json({ error: 'not-found', ring: slug, member }, 404);
35+
return json({ ring: loaded.ring.slug, member: memberEntry(row) });
36+
}
37+
38+
/**
39+
* @param {Request} req
40+
* @param {{ params: Promise<{ slug: string, member: string }> }} ctx
41+
*/
42+
export async function PUT(req, { params }) {
43+
const { slug, member } = await params;
44+
const client = db();
45+
46+
const ring = await webrings.ringBySlug(client, slug.toLowerCase());
47+
const row = ring ? await webrings.memberBySlug(client, ring.slug, member.toLowerCase()) : null;
48+
if (!ring || !row) return json({ error: 'not-found', ring: slug, member }, 404);
49+
50+
const caller = await callerOf(req);
51+
const verdict = memberEditVerdict(caller, {
52+
profiles: await profilesForFeed(row.feed_id),
53+
admins: adminEmails(),
54+
});
55+
if (!verdict.ok) return json({ error: verdict.error }, verdict.status);
56+
57+
const text = await req.text();
58+
if (text.length > BODY_LIMIT) return json({ error: `body over ${BODY_LIMIT} bytes` }, 413);
59+
let body;
60+
try {
61+
body = JSON.parse(text || '{}');
62+
} catch {
63+
return json({ error: 'bad-json' }, 400);
64+
}
65+
const patch = madeByPatch(body);
66+
if (!patch.ok) return json({ error: patch.error }, 400);
67+
68+
const saved = await webrings.setMemberMadeBy(client, ring.slug, row.feed_id, {
69+
madeBy: patch.madeBy,
70+
disclosure: patch.disclosure,
71+
source: verdict.source,
72+
});
73+
forgetRing(ring.slug);
74+
75+
return json({ ok: true, ring: ring.slug, source: verdict.source, member: memberEntry(saved ?? row) });
76+
}
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
import { siteUrl } from '../../../../lib/db.js';
2+
import { loadRing } from '../../../../lib/rings.js';
3+
import { hopUrl, ringFile } from '../../../../lib/openwebring.js';
4+
import { json } from '../../authors/route.js';
5+
6+
export const dynamic = 'force-dynamic';
7+
8+
/**
9+
* One ring, as JSON: the ring file plus where its pages are.
10+
*
11+
* @param {Request} req
12+
* @param {{ params: Promise<{ slug: string }> }} ctx
13+
*/
14+
export async function GET(req, { params }) {
15+
const { slug } = await params;
16+
const loaded = await loadRing(slug);
17+
if (!loaded) return json({ error: 'not-found', slug }, 404);
18+
19+
const base = siteUrl();
20+
const file = ringFile({ base, ring: loaded.ring, members: loaded.members });
21+
22+
return json({
23+
...file,
24+
members: file.members.map((m) => ({
25+
...m,
26+
page: `${base}/${encodeURIComponent(String(m.slug))}`,
27+
next: hopUrl(base, loaded.ring.slug, 'next', String(m.url)),
28+
previous: hopUrl(base, loaded.ring.slug, 'previous', String(m.url)),
29+
})),
30+
active: loaded.ring.active_count,
31+
page: file.ring.url,
32+
opml: `${file.ring.url}/opml`,
33+
random: hopUrl(base, loaded.ring.slug, 'random'),
34+
join: `${file.ring.url}#join`,
35+
});
36+
}
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
import { webrings } from '@rssamplifier/db';
2+
3+
import { db, siteUrl } from '../../../lib/db.js';
4+
import { SPEC, hostEntry } from '../../../lib/openwebring.js';
5+
import { json } from '../authors/route.js';
6+
7+
export const dynamic = 'force-dynamic';
8+
9+
/**
10+
* The rings this site hosts, as JSON.
11+
*
12+
* The same entries as /.well-known/openwebring.json, with the page and the
13+
* API address of each ring added, and the active count beside the listed
14+
* count so a caller can see how much of a ring is live without fetching
15+
* every ring file.
16+
*/
17+
export async function GET() {
18+
const base = siteUrl();
19+
const rings = await webrings.listRings(db());
20+
21+
return json({
22+
total: rings.length,
23+
host: `${base}/.well-known/openwebring.json`,
24+
spec: SPEC,
25+
rings: rings.map((ring) => ({
26+
...hostEntry({ base, ring }),
27+
active: ring.active_count,
28+
kind: ring.kind,
29+
...(ring.topic_slug ? { topic: `${base}/topics/${encodeURIComponent(ring.topic_slug)}` } : {}),
30+
api: `${base}/api/rings/${encodeURIComponent(ring.slug)}`,
31+
})),
32+
});
33+
}

‎apps/web/src/app/layout.jsx‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -304,7 +304,11 @@ export default function RootLayout({ children }) {
304304
<p>
305305
Machine-readable: <a href="/mcp">MCP server</a> · <a href="/cli">CLI</a> ·{' '}
306306
<a href="/api/feeds">JSON API</a> · <a href="/opml">OPML</a> ·{' '}
307-
<a href="/llms.txt">llms.txt</a> · <a href="/crawlstats">Crawler status</a>
307+
<a href="/llms.txt">llms.txt</a> · <a href="/crawlstats">Crawler status</a> ·{' '}
308+
{/* Here rather than in Browse: a ring is a thing another site
309+
links into and a program reads, and its index page is the
310+
human face of /.well-known/openwebring.json. */}
311+
<a href="/ring">Webrings</a>
308312
</p>
309313
{/* The install command itself, not just a link to it. Somebody who
310314
scrolled to the bottom of a feed page is exactly the person who
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
import { hopResponse } from '../../../../../lib/rings.js';
2+
3+
export const dynamic = 'force-dynamic';
4+
5+
/**
6+
* /ring/<slug>/<member>/next: the hop with the member named in the path,
7+
* the shape Hotline and the older IndieWeb rings used. No parameter and no
8+
* Referer needed.
9+
*
10+
* @param {Request} req
11+
* @param {{ params: Promise<{ slug: string, member: string }> }} ctx
12+
*/
13+
export async function GET(req, { params }) {
14+
const { slug, member } = await params;
15+
return hopResponse(req, { slug, memberSlug: member, kind: 'next' });
16+
}

0 commit comments

Comments
 (0)