Add sync-repos workflow to auto-discover repos with arch-docs

[jonathanpopham]

Context

There are 53 repos in the org with arch-docs.yml workflows, but only 33 are listed in repos.yaml. Missing repos include: next.js, django, transformers, supabase, deno, bun, go, rust, spring-boot, spring-framework, guava, elasticsearch, kafka, flink, pytorch, nest, prisma, webpack, grafana, volt, toolate-tv, openclaw.

Requirements

Create .github/workflows/sync-repos.yml that:

1. Runs on schedule — every 6 hours (cron: '0 */6 * *') + workflow_dispatch for manual trigger
2. Scans all org repos — use gh api --paginate "orgs/supermodeltools/repos?per_page=100&type=all" to list all repos
3. Checks for arch-docs — for each repo, check if .github/workflows/arch-docs.yml exists
4. Skips repos already in repos.yaml — grep for name: {repo_name} to avoid duplicates
5. Skips supermodeltools.github.io itself
6. Adds missing repos to repos.yaml using yq:
   • Forks (.fork == true) → append to .categories[1].repos (Community)
   • Org-native repos → append to .categories[0].repos (Supermodel Open Source)
   • Pull description from GitHub API (.description)
   • Map .language to pill/pill_class:
     • JavaScript/TypeScript → pill-blue
     • Python → pill-green
     • Go/Rust/C/C++ → pill-accent
     • Java/Kotlin/Scala/Ruby/Swift → pill-orange
     • Shell/HCL → DevOps, pill-green
   • For forks, include upstream: {parent.full_name}
7. Commit and push if any repos were added — use supermodel-bot as committer, message: Sync N repo(s) discovered with arch-docs
8. Use concurrency group add-repo with cancel-in-progress: false

Secrets needed

The workflow needs a BOT_TOKEN secret (GitHub PAT) with repo scope to push commits that trigger the build-index.yml workflow. The default GITHUB_TOKEN won't trigger downstream workflows.

Note: If BOT_TOKEN doesn't exist yet, document that it needs to be created and use ${{ secrets.GITHUB_TOKEN }} as fallback with a comment explaining the limitation.

Reference

A previous version of this workflow existed in commit 1a7a04f — you can reference it with git show 1a7a04f -- .github/workflows/sync-repos.yml.

Acceptance criteria

• Workflow runs successfully on manual trigger
• Missing repos are added to correct categories in repos.yaml
• Fork repos get upstream field set
• No duplicate entries created
• Push triggers build-index.yml to rebuild the site

@claude

[jonathanpopham]

@claude Please implement this issue. Create a PR with the sync-repos workflow.

[jonathanpopham]

@claude Please implement this issue.

[claude]

Claude finished @jonathanpopham's task in 4m 22s —— View job

---

Working on Issue #8: Add sync-repos workflow

• Gather context and understand request
• Read existing repo structure and reference commit
• Draft the sync-repos workflow
• Attempt to create workflow file ⚠️ — push blocked (see note below)

---

⚠️ Limitation: Workflow file push blocked

I'm unable to push .github/workflows/sync-repos.yml — the GitHub App token used by this automation does not have the workflows permission required to create or modify files in .github/workflows/. This is a known limitation.

To implement this: copy the workflow below and create the file manually, or via a PAT-authenticated push.

---

Complete workflow: .github/workflows/sync-repos.yml

name: Sync Repos

on:
  schedule:
    - cron: '0 */6 * * *'
  workflow_dispatch:

concurrency:
  group: add-repo
  cancel-in-progress: false

permissions:
  contents: write

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          # BOT_TOKEN (PAT with 'repo' scope) is required so that the push
          # triggers the build-index.yml workflow. The default GITHUB_TOKEN
          # cannot trigger downstream workflow runs due to GitHub's security model.
          # To create: Settings → Secrets and variables → Actions → New secret named BOT_TOKEN.
          token: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}

      - name: Install yq
        run: |
          sudo wget -qO /usr/local/bin/yq \
            https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64
          sudo chmod +x /usr/local/bin/yq

      - name: Sync repos with arch-docs
        id: sync
        env:
          GH_TOKEN: ${{ secrets.BOT_TOKEN || secrets.GITHUB_TOKEN }}
        run: |
          set -euo pipefail
          ADDED=0

          # Fetch all org repos, one JSON object per line
          gh api --paginate \
            "orgs/supermodeltools/repos?per_page=100&type=all" \
            --jq '.[] | @json' > /tmp/repos.ndjson

          while IFS= read -r repo_json; do
            REPO_NAME=$(echo "$repo_json" | jq -r '.name')
            IS_FORK=$(echo "$repo_json" | jq -r '.fork')
            DESCRIPTION=$(echo "$repo_json" | jq -r '.description // ""')
            LANGUAGE=$(echo "$repo_json" | jq -r '.language // ""')
            PARENT=$(echo "$repo_json" | jq -r '.parent.full_name // ""')

            # Skip this repository itself
            if [ "$REPO_NAME" = "supermodeltools.github.io" ]; then
              continue
            fi

            # Skip if already listed in repos.yaml
            if grep -qE "^\s+name: ${REPO_NAME}$" repos.yaml; then
              continue
            fi

            # Check for arch-docs workflow — skip if not present
            if ! gh api "repos/supermodeltools/${REPO_NAME}/contents/.github/workflows/arch-docs.yml" \
                >/dev/null 2>&1; then
              continue
            fi

            # Map language to pill_class
            case "$LANGUAGE" in
              JavaScript|TypeScript)        PILL_CLASS="pill-blue" ;;
              Python)                       PILL_CLASS="pill-green" ;;
              Go|Rust|C|"C++")             PILL_CLASS="pill-accent" ;;
              Java|Kotlin|Scala|Ruby|Swift) PILL_CLASS="pill-orange" ;;
              Shell|HCL)                    PILL_CLASS="pill-green" ;;
              *)                            PILL_CLASS="" ;;
            esac

            # Map language to pill label (Shell/HCL → DevOps)
            case "$LANGUAGE" in
              Shell|HCL) PILL="DevOps" ;;
              "")        PILL="" ;;
              *)         PILL="$LANGUAGE" ;;
            esac

            echo "Adding: ${REPO_NAME} (fork=${IS_FORK}, lang=${LANGUAGE})"

            if [ "$IS_FORK" = "true" ]; then
              # Append to Community category (index 1)
              REPO_NAME="$REPO_NAME" \
              DESCRIPTION="$DESCRIPTION" \
              PILL="$PILL" \
              PILL_CLASS="$PILL_CLASS" \
              UPSTREAM="$PARENT" \
              yq -i '.categories[1].repos += [{
                "name": env(REPO_NAME),
                "upstream": env(UPSTREAM),
                "description": env(DESCRIPTION),
                "pill": env(PILL),
                "pill_class": env(PILL_CLASS)
              }]' repos.yaml
            else
              # Append to Supermodel Open Source category (index 0)
              REPO_NAME="$REPO_NAME" \
              DESCRIPTION="$DESCRIPTION" \
              PILL="$PILL" \
              PILL_CLASS="$PILL_CLASS" \
              yq -i '.categories[0].repos += [{
                "name": env(REPO_NAME),
                "description": env(DESCRIPTION),
                "pill": env(PILL),
                "pill_class": env(PILL_CLASS)
              }]' repos.yaml
            fi

            ADDED=$((ADDED + 1))
          done < /tmp/repos.ndjson

          echo "added=${ADDED}" >> "$GITHUB_OUTPUT"
          echo "Sync complete: ${ADDED} repo(s) added"

      - name: Commit and push
        if: steps.sync.outputs.added != '0'
        run: |
          git config user.name "supermodel-bot"
          git config user.email "bot@supermodeltools.com"
          git add repos.yaml
          git commit -m "Sync ${{ steps.sync.outputs.added }} repo(s) discovered with arch-docs"
          git push

---

Design notes

• Dedup: grep -qE "^\s+name: ${REPO_NAME}$" matches the YAML indented name field, preventing duplicates.
• arch-docs check: gh api repos/supermodeltools/{repo}/contents/.github/workflows/arch-docs.yml — 404 = skip.
• Language mapping: shell case statement maps GitHub's .language field to pill / pill_class.
• Safe yq updates: uses env(VAR) in yq expressions to avoid quoting/injection issues with descriptions that contain special characters.
• BOT_TOKEN fallback: uses secrets.BOT_TOKEN || secrets.GITHUB_TOKEN — works but note that without BOT_TOKEN, the push won't trigger build-index.yml.
• Committer: supermodel-bot as specified.

[jonathanpopham]

@claude The workflow scope issue is now fixed (BOT_TOKEN with workflow scope added). Please implement this issue — create a PR with the sync-repos workflow.

[jonathanpopham]

Claude finished @jonathanpopham's task in 1m 47s —— View job • claude/issue-8-20260303-2011 • [Create PR ➔](https://github.com/supermodeltools/supermodeltools.github.io/compare/main...claude/issue-8-20260303-2011?quick_pull=1&title=Issue%20%238%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%238%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

Implementing sync-repos workflow

• Gather context and understand request
• Check reference commit and existing repo structure
• Create .github/workflows/sync-repos.yml
• Commit and push
• Open PR

---

Done

Created .github/workflows/sync-repos.yml and opened a PR:

PR #15: Add sync-repos workflow to auto-discover repos with arch-docs

What was implemented

• Schedule: every 6 hours (cron: '0 */6 * * *') + workflow_dispatch
• Concurrency: group add-repo, cancel-in-progress: false
• Org scan: gh api --paginate "orgs/supermodeltools/repos?per_page=100&type=all"
• arch-docs check: HEAD request to repos/supermodeltools/{repo}/contents/.github/workflows/arch-docs.yml — 404 = skip
• Dedup: grep -qE "^\s+name: ${REPO_NAME}$" on repos.yaml
• Self-skip: skips supermodeltools.github.io itself
• Category routing: forks → categories[1] (Community), org-native → categories[0] (Supermodel Open Source)
• Language mapping: JS/TS → pill-blue, Python → pill-green, Go/Rust/C/C++ → pill-accent, Java/Kotlin/Scala/Ruby/Swift → pill-orange, Shell/HCL → DevOps pill-green
• Upstream field: set for forks from .parent.full_name
• Commit: supermodel-bot committer, message Sync N repo(s) discovered with arch-docs
• Token: uses BOT_TOKEN (with repo + workflow scopes) with GITHUB_TOKEN fallback