Compare commits

..

98 Commits

Author SHA1 Message Date
Arunavo Ray
79e0086a72 sso: polyfill buffer for browser 2025-10-22 18:13:36 +05:30
Arunavo Ray
dc340666ef v3.8.7 2025-10-22 17:54:25 +05:30
ARUNAVO RAY
8b50a07c68 Merge pull request #124 from RayLabsHQ/fix/sso-stability
Fix/sso stability
2025-10-22 17:54:01 +05:30
Arunavo Ray
7dab4fb1d5 sso: drop legacy migration helper 2025-10-22 16:34:37 +05:30
Arunavo Ray
847823bbf8 sso: normalize provider config via discovery 2025-10-22 16:33:33 +05:30
Arunavo Ray
e4e54722cf auth: bump better auth beta 2025-10-22 16:28:59 +05:30
Arunavo Ray
1eddbad908 v3.8.6 2025-10-22 14:56:05 +05:30
ARUNAVO RAY
a7083beff5 Merge pull request #123 from RayLabsHQ/issue-84-archive-loop-fix
Issue 84 archive loop fix
2025-10-22 14:55:25 +05:30
Arunavo Ray
b21cd0b866 ui: align cleanup retention controls 2025-10-22 14:49:09 +05:30
Arunavo Ray
df644be769 cleanup: improve orphaned repo handling 2025-10-22 14:41:54 +05:30
Arunavo Ray
204869fa3e downgraded bun 2025-10-22 13:46:30 +05:30
Arunavo Ray
e470256475 v3.8.5 2025-10-22 09:40:32 +05:30
ARUNAVO RAY
b65c360d61 Merge pull request #121 from RayLabsHQ/fix-release-sync
fix: mirror releases during sync
2025-10-22 09:36:53 +05:30
Arunavo Ray
ce46d33d29 updated packages 2025-10-22 09:36:41 +05:30
Arunavo Ray
f63633f97e fix: mirror releases during sync 2025-10-22 09:33:13 +05:30
Arunavo Ray
3b53a29e71 v3.8.4 2025-10-22 09:07:02 +05:30
ARUNAVO RAY
64e73f9ca8 Merge pull request #120 from RayLabsHQ/fix-org-destination-override
fix: Custom organization mirror destination issue
2025-10-22 09:06:13 +05:30
Arunavo Ray
7d23894e5f fix: Custom organization mirror destination issue 2025-10-22 09:03:27 +05:30
ARUNAVO RAY
8f2a4683c1 Merge pull request #119 from RayLabsHQ/fix/duplicate-repos-issue-115
Fix/duplicate repos issue 115
2025-10-22 08:59:07 +05:30
Arunavo Ray
b5323ff8b4 chore: remove duplicate fix summary 2025-10-22 08:44:22 +05:30
Arunavo Ray
7fee2adb51 lib: harden mirror idempotency 2025-10-22 08:43:38 +05:30
ARUNAVO RAY
af139ecb2d Merge pull request #114 from RayLabsHQ/dependabot/npm_and_yarn/www/npm_and_yarn-82a5d800d0
Bump astro from 5.14.1 to 5.14.3 in /www in the npm_and_yarn group across 1 directory
2025-10-11 11:40:53 +05:30
dependabot[bot]
fb827724b6 Bump astro in /www in the npm_and_yarn group across 1 directory
Bumps the npm_and_yarn group with 1 update in the /www directory: [astro](https://github.com/withastro/astro/tree/HEAD/packages/astro).


Updates `astro` from 5.14.1 to 5.14.3
- [Release notes](https://github.com/withastro/astro/releases)
- [Changelog](https://github.com/withastro/astro/blob/main/packages/astro/CHANGELOG.md)
- [Commits](https://github.com/withastro/astro/commits/astro@5.14.3/packages/astro)

---
updated-dependencies:
- dependency-name: astro
  dependency-version: 5.14.3
  dependency-type: direct:production
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-10-11 00:40:10 +00:00
ARUNAVO RAY
2812b576d0 Merge pull request #112 from V-Paranoiaque/fix-custom-image-tag
Fix when using a custom tag
2025-10-05 18:22:47 +05:30
Virgil R.
347188f43d Fix when using a custom tag 2025-10-04 14:52:42 +02:00
Arunavo Ray
56bee451de [www] : updated packages 2025-10-03 09:53:33 +05:30
Arunavo Ray
0e9d54b517 fix: add backward compatibility for skipStarredIssues field
Since githubConfig is stored as JSON in the database (not individual columns),
no database migration is needed. However, we need to handle reading old configs
that still use the 'skipStarredIssues' field name.

Changes:
- Added skipStarredIssues as optional field in Zod schema (marked deprecated)
- Updated config mapper to check both starredCodeOnly and skipStarredIssues
- Old configs will continue to work seamlessly
- New configs will use starredCodeOnly field name

This ensures zero-downtime upgrades for existing installations.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 09:44:38 +05:30
Arunavo Ray
7a04665b70 v3.8.3 2025-10-03 09:24:28 +05:30
Arunavo Ray
3a3ff314e0 refactor: rename skipStarredIssues to starredCodeOnly
The previous name 'skipStarredIssues' was misleading as it now skips ALL
metadata (not just issues) for starred repositories. The new name
'starredCodeOnly' better reflects the actual behavior - mirroring only
source code for starred repos.

Changes:
- Renamed skipStarredIssues → starredCodeOnly in all files
- Updated UI label from "Don't fetch issues" to "Code-only mode"
- Updated description to clarify it skips ALL metadata types:
  issues, PRs, labels, milestones, wiki, and releases
- Updated database schema, types, config mapper, and all components
- Updated Helm charts, CI configs, and documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 09:22:18 +05:30
Arunavo Ray
fed74ee901 fix: apply skipStarredIssues flag to releases mirroring
Extended the skipStarredIssues flag to also skip releases for starred repos
when "code-only" mode is enabled. Previously, releases were still being
mirrored even when lightweight mode was selected.

Now starred repos with "code-only" mode will skip:
- Issues ✓
- Pull requests ✓
- Labels ✓
- Milestones ✓
- Wiki ✓
- Releases ✓ (this fix)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 09:18:35 +05:30
Arunavo Ray
85ea502276 fix: apply skipStarredIssues flag to all metadata types
Previously, the skipStarredIssues flag (Lightweight mode for starred repos)
only applied to issues. This caused starred repos to mirror all metadata
(pull requests, labels, milestones, wiki) even when "code-only" was selected.

Fixed by applying the skipStarredIssues check to:
- Pull requests mirroring
- Labels mirroring
- Milestones mirroring
- Wiki mirroring (in migration payload)

Now starred repos with "code-only" mode truly mirror only source code,
skipping all metadata as intended.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-10-03 09:13:58 +05:30
Arunavo Ray
ffb7bd3cb0 v3.8.2 2025-10-02 21:19:32 +05:30
ARUNAVO RAY
b39d7a2179 Merge pull request #107 from RayLabsHQ/fix/issue-97-migration-duplicates
fix: resolve migration 0005 duplicate constraint failure (#97)
2025-10-02 07:45:42 +05:30
Arunavo Ray
bf99a95dc6 ci: add more paths to trigger Docker builds
- Added docker-entrypoint.sh to trigger paths
- Added drizzle/** for database migrations
- Added scripts/** for database management scripts
- Added src/** for source code changes

This ensures Docker images are rebuilt when critical runtime
files change, not just package dependencies.
2025-10-01 08:02:56 +05:30
Arunavo Ray
2ea917fdaa fix: resolve migration 0005 duplicate constraint failure (#97)
**Problem:**
- Users upgrading to v3.7.2 encountered database migration failures
- Migration 0005 tried to add unique index without handling existing duplicates
- Hybrid initialization (manual SQL + Drizzle) caused schema inconsistencies
- Error: "UNIQUE constraint failed: repositories.user_id, repositories.full_name"

**Solution:**

1. **Fixed Migration 0005:**
   - Added deduplication step before creating unique index
   - Removes duplicate (user_id, full_name) entries, keeping most recent
   - Safely creates unique constraint after cleanup

2. **Removed Hybrid Database Initialization:**
   - Eliminated 154 lines of manual SQL from docker-entrypoint.sh
   - Now uses Drizzle exclusively for schema management
   - Single source of truth prevents schema drift
   - Migrations run automatically via src/lib/db/index.ts

**Testing:**
-  Fresh database initialization works
-  Duplicate deduplication verified
-  Unique constraint properly enforced
-  All 6 migrations apply cleanly

**Changes:**
- docker-entrypoint.sh: Removed manual table creation SQL
- drizzle/0005_polite_preak.sql: Added deduplication before index creation

Fixes #97
2025-10-01 07:57:16 +05:30
Arunavo Ray
b841057f1a updated packages 2025-10-01 07:29:39 +05:30
ARUNAVO RAY
d588ce91b4 Merge pull request #106 from RayLabsHQ/alert-autofix-33
Potential fix for code scanning alert no. 33: Workflow does not contain permissions
2025-10-01 07:06:58 +05:30
ARUNAVO RAY
553396483e Potential fix for code scanning alert no. 33: Workflow does not contain permissions
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2025-10-01 07:06:06 +05:30
ARUNAVO RAY
ebeabdb4fc Merge pull request #105 from RayLabsHQ/fix/forgejo-12-private-repos
fix: Forgejo 12 compatibility - use separate auth fields for private repos (#102)
2025-10-01 07:02:30 +05:30
ARUNAVO RAY
ff209a6376 Merge pull request #101 from V-Paranoiaque/helm-chart
Helm chart to deploy on Kubernetes
2025-10-01 07:01:19 +05:30
Arunavo Ray
096e0c03ac images now tagged lowercase in comments 2025-09-30 23:43:27 +05:30
Virgil R.
63f20a7f04 Update helm-test.yml 2025-09-30 20:07:48 +02:00
Arunavo Ray
34f741beef fix: Forgejo 12 compatibility - use separate auth fields for private repos (#102)
## Problem
Forgejo 12.0+ rejects migration API calls with credentials embedded in URLs,
causing HTTP 422 errors when mirroring private GitHub repositories.

## Root Cause
Breaking security change in Forgejo 12.0 (July 2025) enforces credential
separation to prevent accidental exposure in logs/errors. Previous versions
(Forgejo 11.x, Gitea 1.x) accepted embedded credentials.

## Solution
- Use separate `auth_username` and `auth_token` fields instead of embedding
  credentials in clone URLs
- Set `auth_username` to "oauth2" for GitHub token authentication
- Pass GitHub token via `auth_token` field

## Changes
- src/lib/gitea.ts:
  - mirrorGithubRepoToGitea(): Use separate auth fields for private repos
  - mirrorGitHubRepoToGiteaOrg(): Use separate auth fields for private repos

- .github/workflows/docker-build.yml:
  - Enable PR image building and pushing to GHCR
  - Tag PR images as pr-<number> for easy testing
  - Add automated PR comment with image details and testing instructions
  - Separate load step for security scanning

## Backward Compatibility
 Works with Forgejo 12.0+
 Works with Forgejo 11.x and earlier
 Works with Gitea 1.x

## Testing
Public repos:  Working (no auth needed)
Private repos:  Fixed (separate auth fields)

Fixes #102
2025-09-30 23:12:33 +05:30
V-Paranoiaque
1f98f441f3 Fix ingress + improve testing 2025-09-27 18:28:10 +02:00
V-Paranoiaque
9c1ac76ff9 Fix annotations 2025-09-27 15:18:52 +02:00
V-Paranoiaque
cf5027bafc Fix CLEANUP_RETENTION_DAYS 2025-09-27 15:15:42 +02:00
V-Paranoiaque
6fd2774d43 Fix MIRROR_STARRED var 2025-09-27 15:13:54 +02:00
V-Paranoiaque
8f379baad4 Improve CI/CD 2025-09-27 10:34:20 +02:00
V-Paranoiaque
91fa3604b6 Add some basic CICD for testing 2025-09-27 10:24:18 +02:00
Virgil R.
c0fff30fcb Create README.md 2025-09-27 10:12:29 +02:00
Virgil R.
18de63d192 Update deployment.yaml 2025-09-27 10:04:42 +02:00
V-Paranoiaque
1fe20c3e54 GITHUB_TYPE env var 2025-09-24 22:07:15 +02:00
V-Paranoiaque
7386b54a46 Fix env vars 2025-09-24 21:12:16 +02:00
V-Paranoiaque
432a2bc54d Add missing podSecurityContext 2025-09-24 20:47:54 +02:00
V-Paranoiaque
f9d18f34ab Fix image tag 2025-09-24 20:41:24 +02:00
V-Paranoiaque
cd86a09bbd Minor fixes 2025-09-24 20:29:14 +02:00
V-Paranoiaque
1e2c1c686d Add volumes 2025-09-21 21:15:50 +02:00
V-Paranoiaque
f701574e67 Move NODE_ENV to the configmap 2025-09-21 15:15:58 +02:00
V-Paranoiaque
4528be8cc6 Too many spaces 2025-09-21 12:29:02 +02:00
V-Paranoiaque
80fd43ef42 More features 2025-09-21 12:26:26 +02:00
V-Paranoiaque
3c52fe58aa Missing space 2025-09-21 10:23:48 +02:00
V-Paranoiaque
319e7925ff First commit TBT 2025-09-20 22:59:22 +02:00
Arunavo Ray
5add8766a4 fix(scheduler,config): preserve ENV schedule; add AUTO_MIRROR_REPOS auto-mirroring
- Prevent Automation UI from overriding schedule:
      - mapDbScheduleToUi now parses intervals robustly (cron/duration/seconds) via parseInterval
      - mapUiScheduleToDb merges with existing config and stores interval as seconds (no lossy cron conversion)
      - /api/config passes existing scheduleConfig to preserve ENV-sourced values
      - schedule-sync endpoint uses parseInterval for nextRun calculation
  - Add AUTO_MIRROR_REPOS support and scheduled auto-mirror phase:
      - scheduleConfig schema includes autoImport and autoMirror
      - env-config-loader reads AUTO_MIRROR_REPOS and carries through to DB
      - scheduler auto-mirrors imported/pending/failed repos when autoMirror is enabled before regular sync
      - docker-compose and ENV docs updated with AUTO_MIRROR_REPOS
  - Tests pass and build succeeds
2025-09-14 08:31:31 +05:30
Arunavo Ray
6ce70bb5bf chore(version): bump to 3.7.1\n\ncleanup: attempt fix for orphaned repo archiving (refs #84)\n- Sanitize mirror rename to satisfy AlphaDashDot; timestamped fallback\n- Resolve Gitea owner robustly via mirroredLocation/strategy; verify presence\n- Add 'archived' status to Zod enums; set isArchived on archive\n- Update CHANGELOG entry without closing keyword 2025-09-14 07:53:36 +05:30
Arunavo Ray
f3aae2ec94 fix for repo name collison 2025-09-14 00:13:13 +05:30
Arunavo Ray
46d5ec46fc Updated deisgn for 'Duplicate collision strategy' 2025-09-13 23:54:14 +05:30
Arunavo Ray
0caa53b67f v3.7.0 2025-09-13 23:39:50 +05:30
Arunavo Ray
18ecdbc252 fix(sync): batch inserts + normalize nulls to avoid SQLite param mismatch
- Batch repository inserts with dynamic sizing under SQLite 999-param limit
- Normalize undefined → null to keep multi-row insert shapes consistent
- De-duplicate owned + starred repos by fullName (prefer starred variant)
- Enforce uniqueness via (user_id, full_name) + onConflictDoNothing
- Handle starred name collisions (suffix/prefix) across mirror + metadata
- Add repo-utils helpers + tests; guard Octokit.plugin in tests
- Remove manual unique index from entrypoint; rely on drizzle-kit migrations
2025-09-13 23:38:50 +05:30
Arunavo Ray
51a6c8ca58 Added product hunt badge on website 2025-09-12 01:44:13 +05:30
Arunavo Ray
41b8806268 update packages 2025-09-10 09:49:08 +05:30
ARUNAVO RAY
ac5c7800c1 Merge pull request #93 from RayLabsHQ/dependabot/npm_and_yarn/www/npm_and_yarn-73ea615029
Bump vite from 6.3.5 to 6.3.6 in /www in the npm_and_yarn group across 1 directory
2025-09-10 09:46:02 +05:30
dependabot[bot]
13e7661f07 Bump vite in /www in the npm_and_yarn group across 1 directory
Bumps the npm_and_yarn group with 1 update in the /www directory: [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite).


Updates `vite` from 6.3.5 to 6.3.6
- [Release notes](https://github.com/vitejs/vite/releases)
- [Changelog](https://github.com/vitejs/vite/blob/v6.3.6/packages/vite/CHANGELOG.md)
- [Commits](https://github.com/vitejs/vite/commits/v6.3.6/packages/vite)

---
updated-dependencies:
- dependency-name: vite
  dependency-version: 6.3.6
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-09-10 02:49:30 +00:00
Arunavo Ray
37e5b68bd5 Added Github API rate limiting
- Implemented comprehensive GitHub API rate limit handling:
    - Integrated @octokit/plugin-throttling for automatic retry with exponential backoff
    - Added RateLimitManager service to track and enforce rate limits
    - Store rate limit status in database for persistence across restarts
    - Automatic pause and resume when limits are exceeded
    - Proper user identification for 5000 req/hr authenticated limit (vs 60 unauthenticated)

  - Improved rate limit UI/UX:
    - Removed intrusive rate limit card from dashboard
    - Toast notifications only at critical thresholds (80% and 100% usage)
    - All rate limit events logged for debugging

  - Optimized for GitHub's API constraints:
    - Reduced default batch size from 10 to 5 repositories
    - Added documentation about GitHub's 100 concurrent request limit
    - Better handling of repositories with many issues/PRs
2025-09-09 11:14:43 +05:30
Arunavo Ray
89ca5abe7d fix: resolve SQLite field mismatch for large starred repo imports (#90)
- Add missing database fields (language, description, mirroredLocation, destinationOrg) to repository operations
  - Add missing organization fields (publicRepositoryCount, privateRepositoryCount, forkRepositoryCount) to schema
  - Update GitRepo interface to include all required database fields
  - Fix GitHub data fetching functions to map all fields correctly
  - Update all sync endpoints (main, repository, organization, scheduler) to handle new fields

  This fixes the "SQLite query expected X values, received Y" error when importing
  large numbers (4.6k+) of starred repositories by ensuring all database fields
  are properly mapped from GitHub API responses through to database insertion.
2025-09-09 09:56:18 +05:30
Arunavo Ray
2b78a6a4a8 v3.5.4 2025-09-07 19:11:50 +05:30
Arunavo Ray
c2f6e73054 Testing Authentik SSO Issues 2025-09-07 19:09:00 +05:30
Arunavo Ray
c4b353aae8 Added docs around scheduling using corn 2025-09-07 16:51:51 +05:30
Arunavo Ray
4a54cf9009 v3.5.3 2025-09-07 16:29:43 +05:30
Arunavo Ray
fab4efd93a Auto-start on boot 2025-09-07 16:29:23 +05:30
Arunavo Ray
9f21cd6b1a Addressing concerns of Issue #85 and #86 2025-09-07 15:25:48 +05:30
Arunavo Ray
9ef6017a23 v3.5.2 2025-09-07 13:55:43 +05:30
Arunavo Ray
502796371f Attempt to address #84 2025-09-07 13:55:20 +05:30
Arunavo Ray
b956b71c5f Fixed #87 where the Release Notes was missing 2025-09-07 13:14:41 +05:30
Arunavo Ray
26b82e0f65 Added AGENTS.md 2025-09-07 11:46:14 +05:30
Arunavo Ray
7c124a37d7 v3.5.1 2025-08-30 00:47:59 +05:30
Arunavo Ray
3e14edc571 fixed default overide 2025-08-30 00:47:33 +05:30
Arunavo Ray
a188869cae "Automatic Mirroring" changed to "Automatic Syncing" 2025-08-30 00:37:56 +05:30
Arunavo Ray
afac3b5ddc UI tweek 2025-08-29 21:16:19 +05:30
Arunavo Ray
2ce4bb4373 update env doc 2025-08-29 20:43:49 +05:30
Arunavo Ray
5c9a3afaae updates to auth url 2025-08-29 20:43:25 +05:30
Arunavo Ray
de4e111095 type fix 2025-08-29 20:42:56 +05:30
Arunavo Ray
8c4d9508c7 Add provider modal optimised 2025-08-29 19:17:40 +05:30
Arunavo Ray
921eb5e07d util 2025-08-29 19:08:48 +05:30
Arunavo Ray
ac1b09f7a1 UI updates 2025-08-29 19:08:39 +05:30
Arunavo Ray
9ee67ce77d made time more user readable 2025-08-29 18:32:22 +05:30
Arunavo Ray
92db61a2c9 v3.5.0 2025-08-29 18:11:49 +05:30
Arunavo Ray
cbf6e11de3 Env var updates 2025-08-29 18:11:26 +05:30
Arunavo Ray
18855f09c4 Imporved a bunch of things in Mirror and sync Automation 2025-08-29 17:49:44 +05:30
102 changed files with 12148 additions and 2354 deletions

View File

@@ -18,6 +18,7 @@ DATABASE_URL=sqlite://data/gitea-mirror.db
# Generate with: openssl rand -base64 32
BETTER_AUTH_SECRET=change-this-to-a-secure-random-string-in-production
BETTER_AUTH_URL=http://localhost:4321
# PUBLIC_BETTER_AUTH_URL=https://your-domain.com # Optional: Set this if accessing from different origins (e.g., IP and domain)
# ENCRYPTION_SECRET=optional-encryption-key-for-token-encryption # Generate with: openssl rand -base64 48
# ===========================================
@@ -94,6 +95,7 @@ DOCKER_TAG=latest
# Release and Metadata
# MIRROR_RELEASES=false # Mirror GitHub releases
# RELEASE_LIMIT=10 # Maximum number of releases to mirror per repository
# MIRROR_WIKI=false # Mirror wiki content
# Issue Tracking (requires MIRROR_METADATA=true)
@@ -109,8 +111,10 @@ DOCKER_TAG=latest
# ===========================================
# Basic Schedule Settings
# SCHEDULE_ENABLED=false
# SCHEDULE_ENABLED=false # When true, auto-imports and mirrors all repos on startup (v3.5.3+)
# SCHEDULE_INTERVAL=3600 # Interval in seconds or cron expression (e.g., "0 2 * * *")
# GITEA_MIRROR_INTERVAL=8h # Mirror sync interval (5m, 30m, 1h, 8h, 24h, 1d, 7d) - also triggers auto-start
# AUTO_IMPORT_REPOS=true # Automatically discover and import new GitHub repositories during syncs
# DELAY=3600 # Legacy: same as SCHEDULE_INTERVAL, kept for backward compatibility
# Execution Settings
@@ -148,11 +152,11 @@ DOCKER_TAG=latest
# CLEANUP_ENABLED=false
# CLEANUP_RETENTION_DAYS=7 # Days to keep events
# Repository Cleanup
# Repository Cleanup (v3.4.0+)
# CLEANUP_DELETE_FROM_GITEA=false # Delete repos from Gitea
# CLEANUP_DELETE_IF_NOT_IN_GITHUB=true # Delete if not in GitHub - automatically enables cleanup
# CLEANUP_DELETE_IF_NOT_IN_GITHUB=false # Auto-remove repos that no longer exist in GitHub
# CLEANUP_ORPHANED_REPO_ACTION=archive # Options: skip, archive, delete
# CLEANUP_DRY_RUN=true # Test mode without actual deletion
# CLEANUP_DRY_RUN=true # Test mode without actual deletion (set to false for production)
# Protected Repositories (comma-separated)
# CLEANUP_PROTECTED_REPOS=important-repo,critical-project

BIN
.github/assets/logo-new.png vendored Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

59
.github/ci/values-ci.yaml vendored Normal file
View File

@@ -0,0 +1,59 @@
image:
registry: ghcr.io
repository: raylabshq/gitea-mirror
tag: ""
service:
type: ClusterIP
port: 8080
ingress:
enabled: true
className: "nginx"
hosts:
- host: ci.example.com
route:
enabled: true
forceHTTPS: true
domain: ["ci.example.com"]
gateway: "dummy-gw"
gatewayNamespace: "default"
http:
gatewaySection: "http"
https:
gatewaySection: "https"
gitea-mirror:
nodeEnv: production
core:
databaseUrl: "file:data/gitea-mirror.db"
betterAuthSecret: "dummy"
betterAuthUrl: "http://localhost:4321"
betterAuthTrustedOrigins: "http://localhost:4321"
github:
username: "ci-user"
token: "not-used-in-template"
type: "personal"
privateRepositories: true
skipForks: false
starredCodeOnly: false
gitea:
url: "https://gitea.example.com"
token: "not-used-in-template"
username: "ci-user"
organization: "github-mirrors"
visibility: "public"
mirror:
releases: true
wiki: true
metadata: true
issues: true
pullRequests: true
starred: false
automation:
schedule_enabled: true
schedule_interval: "3600"
cleanup:
enabled: true
interval: "2592000"

View File

@@ -85,3 +85,10 @@ If a workflow fails:
- Security vulnerabilities
For persistent issues, consider opening an issue in the repository.
### Helm Test (`helm-test.yml`)
This workflow run on the main branch and pull requests. it:
- Run yamllint to keep the formating unified
- Run helm template with different value files

View File

@@ -10,6 +10,10 @@ on:
- 'package.json'
- 'bun.lock*'
- '.github/workflows/docker-build.yml'
- 'docker-entrypoint.sh'
- 'drizzle/**'
- 'scripts/**'
- 'src/**'
pull_request:
paths:
- 'Dockerfile'
@@ -17,6 +21,10 @@ on:
- 'package.json'
- 'bun.lock*'
- '.github/workflows/docker-build.yml'
- 'docker-entrypoint.sh'
- 'drizzle/**'
- 'scripts/**'
- 'src/**'
schedule:
- cron: '0 0 * * 0' # Weekly security scan on Sunday at midnight
@@ -48,7 +56,6 @@ jobs:
- name: Log into registry
uses: docker/login-action@v3
if: github.event_name != 'pull_request'
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
@@ -89,6 +96,7 @@ jobs:
type=sha,prefix=,suffix=,format=short
type=raw,value=latest,enable={{is_default_branch}}
type=raw,value=${{ steps.tag_version.outputs.VERSION }}
type=ref,event=pr,prefix=pr-
# Build and push Docker image
- name: Build and push Docker image
@@ -97,20 +105,77 @@ jobs:
with:
context: .
platforms: ${{ github.event_name == 'pull_request' && 'linux/amd64' || 'linux/amd64,linux/arm64' }}
push: ${{ github.event_name != 'pull_request' }}
load: ${{ github.event_name == 'pull_request' }}
tags: ${{ github.event_name == 'pull_request' && 'gitea-mirror:scan' || steps.meta.outputs.tags }}
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=gha
cache-to: type=gha,mode=max
# Load image locally for security scanning (PRs only)
- name: Load image for scanning
if: github.event_name == 'pull_request'
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64
load: true
tags: gitea-mirror:scan
cache-from: type=gha
# Wait for image to be available in registry
- name: Wait for image availability
if: github.event_name != 'pull_request'
run: |
echo "Waiting for image to be available in registry..."
sleep 5
# Add comment to PR with image details
- name: Comment PR with image tag
if: github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const prNumber = context.payload.pull_request.number;
const imageTag = `pr-${prNumber}`;
const imagePath = `${{ env.REGISTRY }}/${{ env.IMAGE }}:${imageTag}`.toLowerCase();
const comment = `## 🐳 Docker Image Built Successfully
Your PR image is available for testing:
**Image Tag:** \`${imageTag}\`
**Full Image Path:** \`${imagePath}\`
### Pull and Test
\`\`\`bash
docker pull ${imagePath}
docker run -d -p 3000:3000 --name gitea-mirror-test ${imagePath}
\`\`\`
### Docker Compose Testing
\`\`\`yaml
services:
gitea-mirror:
image: ${imagePath}
ports:
- "3000:3000"
environment:
- BETTER_AUTH_SECRET=your-secret-here
\`\`\`
> 💡 **Note:** PR images are tagged as \`pr-<number>\` and only built for \`linux/amd64\` to speed up CI.
> Production images (\`latest\`, version tags) are multi-platform (\`linux/amd64\`, \`linux/arm64\`).
---
📦 View in [GitHub Packages](https://github.com/${{ github.repository }}/pkgs/container/gitea-mirror)`;
github.rest.issues.createComment({
issue_number: prNumber,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
# Docker Scout comprehensive security analysis
- name: Docker Scout - Vulnerability Analysis & Recommendations
uses: docker/scout-action@v1

61
.github/workflows/helm-test.yml vendored Normal file
View File

@@ -0,0 +1,61 @@
name: Helm Chart CI
permissions:
contents: read
on:
pull_request:
paths:
- 'helm/gitea-mirror/**'
- '.github/workflows/helm-test.yml'
- '.github/ci/values-ci.yaml'
push:
branches: [ main ]
paths:
- 'helm/gitea-mirror/**'
- '.github/workflows/helm-test.yml'
- '.github/ci/values-ci.yaml'
workflow_dispatch:
jobs:
yamllint:
name: Lint YAML
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.x'
- name: Install yamllint
run: pip install --disable-pip-version-check yamllint
- name: Run yamllint
run: |
yamllint -c helm/gitea-mirror/.yamllint helm/gitea-mirror
helm-template:
name: Helm lint & template
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup Helm
uses: azure/setup-helm@v4
with:
version: v3.19.0
- name: Helm lint
run: |
helm lint ./helm/gitea-mirror
- name: Template with defaults
run: |
helm template test ./helm/gitea-mirror > /tmp/render-defaults.yaml
test -s /tmp/render-defaults.yaml
- name: Template with CI values
run: |
helm template test ./helm/gitea-mirror -f .github/ci/values-ci.yaml > /tmp/render-ci.yaml
test -s /tmp/render-ci.yaml
- name: Show a summary
run: |
echo "Rendered with defaults:"
awk 'NR<=50{print} NR==51{print "..."; exit}' /tmp/render-defaults.yaml
echo ""
echo "Rendered with CI values:"
awk 'NR<=50{print} NR==51{print "..."; exit}' /tmp/render-ci.yaml

46
AGENTS.md Normal file
View File

@@ -0,0 +1,46 @@
# Repository Guidelines
## Project Structure & Module Organization
- `src/` app code
- `components/` (React, PascalCase files), `pages/` (Astro/API routes), `lib/` (domain + utilities, kebab-case), `hooks/`, `layouts/`, `styles/`, `tests/`, `types/`, `data/`, `content/`.
- `scripts/` operational TS scripts (DB init, recovery): e.g., `scripts/manage-db.ts`.
- `drizzle/` SQL migrations; `data/` runtime SQLite (`gitea-mirror.db`).
- `public/` static assets; `dist/` build output.
- Key config: `astro.config.mjs`, `tsconfig.json` (alias `@/* → src/*`), `bunfig.toml` (test preload), `.env(.example)`.
## Build, Test, and Development Commands
- Prereq: Bun `>= 1.2.9` (see `package.json`).
- Setup: `bun run setup` install deps and init DB.
- Dev: `bun run dev` start Astro dev server.
- Build: `bun run build` produce `dist/`.
- Preview/Start: `bun run preview` (static preview) or `bun run start` (SSR entry).
- Database: `bun run db:generate|migrate|push|studio` and `bun run manage-db init|check|fix|reset-users`.
- Tests: `bun test` | `bun run test:watch` | `bun run test:coverage`.
- Docker: see `docker-compose.yml` and variants in repo root.
## Coding Style & Naming Conventions
- Language: TypeScript, Astro, React.
- Indentation: 2 spaces; keep existing semicolon/quote style in touched files.
- Components: PascalCase `.tsx` in `src/components/` (e.g., `MainLayout.tsx`).
- Modules/utils: kebab-case in `src/lib/` (e.g., `gitea-enhanced.ts`).
- Imports: prefer alias `@/…` (configured in `tsconfig.json`).
- Do not introduce new lint/format configs; follow current patterns.
## Testing Guidelines
- Runner: Bun test (`bun:test`) with preload `src/tests/setup.bun.ts` (see `bunfig.toml`).
- Location/Names: `**/*.test.ts(x)` under `src/**` (examples in `src/lib/**`).
- Scope: add unit tests for new logic and API route tests for handlers.
- Aim for meaningful coverage on DB, auth, and mirroring paths.
## Commit & Pull Request Guidelines
- Commits: short, imperative, scoped when helpful (e.g., `lib: fix token parsing`, `ui: align buttons`).
- PRs must include:
- Summary, rationale, and testing steps/commands.
- Linked issues (e.g., `Closes #123`).
- Screenshots/gifs for UI changes.
- Notes on DB/migration or .env impacts; update `docs/`/CHANGELOG if applicable.
## Security & Configuration Tips
- Never commit secrets. Copy `.env.example``.env` and fill values; prefer `bun run startup-env-config` to validate.
- SQLite files live in `data/`; avoid committing generated DBs.
- Certificates (if used) reside in `certs/`; manage locally or via Docker secrets.

View File

@@ -58,6 +58,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Updated README with new features
- Enhanced CLAUDE.md with repository status definitions
## [3.7.1] - 2025-09-14
### Fixed
- Cleanup archiving for mirror repositories now works reliably (refs #84; awaiting user confirmation).
- Gitea rejects names violating the AlphaDashDot rule; archiving a mirror now uses a sanitized rename strategy (`archived-<name>`), with a timestamped fallback on conflicts or validation errors.
- Owner resolution during cleanup no longer uses the GitHub owner by mistake. It prefers `mirroredLocation`, falls back to computed Gitea owner via configuration, and verifies location with a presence check to avoid `GetUserByName` 404s.
- Repositories UI crash resolved when cleanup marked repos as archived.
- Added `"archived"` to repository/job status enums, fixing Zod validation errors on the Repositories page.
### Changed
- Archiving logic for mirror repos is non-destructive by design: data is preserved, repo is renamed with an archive marker, and mirror interval is reduced (besteffort) to minimize sync attempts.
- Cleanup service updates DB to `status: "archived"` and `isArchived: true` on successful archive path.
### Notes
- This release addresses the scenario where a GitHub source disappears (deleted/banned), ensuring Gitea backups are preserved even when using `CLEANUP_DELETE_IF_NOT_IN_GITHUB=true` with `CLEANUP_ORPHANED_REPO_ACTION=archive`.
- No database migration required.
## [3.2.6] - 2025-08-09
### Fixed

View File

@@ -193,7 +193,7 @@ export async function POST({ request }: APIContext) {
### Advanced Options (UI Fields)
- **skipForks**: Skip forked repositories (default: false)
- **skipStarredIssues**: Skip issues for starred repositories (default: false) - enables "Lightweight mode" for starred repos
- **starredCodeOnly**: Skip issues for starred repositories (default: false) - enables "Lightweight mode" for starred repos
### Repository Statuses
Repositories can have the following statuses:
@@ -208,6 +208,24 @@ Repositories can have the following statuses:
- **deleting**: Repository being deleted
- **deleted**: Repository deleted
### Scheduling and Synchronization (Issue #72 Fixes)
#### Fixed Issues
1. **Mirror Interval Bug**: Added `mirror_interval` parameter to Gitea API calls when creating mirrors (previously defaulted to 24h)
2. **Auto-Discovery**: Scheduler now automatically discovers and imports new GitHub repositories
3. **Interval Updates**: Sync operations now update existing mirrors' intervals to match configuration
4. **Repository Cleanup**: Integrated automatic cleanup of orphaned repositories (repos removed from GitHub)
#### Environment Variables for Auto-Import
- **AUTO_IMPORT_REPOS**: Set to `false` to disable automatic repository discovery (default: enabled)
#### How Scheduling Works
- **Scheduler Service**: Runs every minute to check for scheduled tasks
- **Sync Interval**: Configured via `GITEA_MIRROR_INTERVAL` or UI (e.g., "8h", "30m", "1d")
- **Auto-Import**: Checks GitHub for new repositories during each scheduled sync
- **Auto-Cleanup**: Removes repositories that no longer exist in GitHub (if enabled)
- **Mirror Interval Update**: Updates Gitea's internal mirror interval during sync operations
### Authentication Configuration
#### SSO Provider Configuration

View File

@@ -1,6 +1,6 @@
# syntax=docker/dockerfile:1.4
FROM oven/bun:1.2.21-alpine AS base
FROM oven/bun:1.2.23-alpine AS base
WORKDIR /app
RUN apk add --no-cache libc6-compat python3 make g++ gcc wget sqlite openssl ca-certificates

View File

@@ -40,7 +40,10 @@ First user signup becomes admin. Configure GitHub and Gitea through the web inte
- 🚫 **Repository ignore** - Mark specific repos to skip
- 🔐 Secure authentication with Better Auth (email/password, SSO, OIDC)
- 📊 Real-time dashboard with activity logs
- ⏱️ Scheduled automatic mirroring with flexible intervals
- ⏱️ Scheduled automatic mirroring with configurable intervals
- 🔄 **Auto-discovery** - Automatically import new GitHub repositories (v3.4.0+)
- 🧹 **Repository cleanup** - Auto-remove repos deleted from GitHub (v3.4.0+)
- 🎯 **Proper mirror intervals** - Respects configured sync intervals (v3.4.0+)
- 🗑️ Automatic database cleanup with configurable retention
- 🐳 Dockerized with multi-arch support (AMD64/ARM64)
@@ -204,25 +207,63 @@ Enable in Settings → Mirror Options → Mirror metadata
- **Automatic Cleanup** - Configure retention period for activity logs
- **Scheduled Sync** - Set custom intervals for automatic mirroring
### Automatic Mirroring
### Automatic Syncing & Synchronization
Gitea Mirror can automatically sync your repositories at regular intervals. There are two ways to configure this:
Gitea Mirror provides powerful automatic synchronization features:
#### Via Web Interface (Recommended)
Navigate to the Configuration page and enable "Automatic Mirroring" with your preferred interval (e.g., every 6 hours, daily, etc.).
#### Features (v3.4.0+)
- **Auto-discovery**: Automatically discovers and imports new GitHub repositories
- **Repository cleanup**: Removes repositories that no longer exist in GitHub
- **Proper intervals**: Mirrors respect your configured sync intervals (not Gitea's default 24h)
- **Smart scheduling**: Only syncs repositories that need updating
- **Auto-start on boot** (v3.5.3+): Automatically imports and mirrors all repositories when `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` is set - no manual clicks required!
#### Via Environment Variables
Set `GITEA_MIRROR_INTERVAL` to automatically enable scheduled mirroring:
#### Configuration via Web Interface (Recommended)
Navigate to the Configuration page and enable "Automatic Syncing" with your preferred interval.
#### Configuration via Environment Variables
**🚀 Set it and forget it!** With these environment variables, Gitea Mirror will automatically:
1. **Import** all your GitHub repositories on startup (no manual import needed!)
2. **Mirror** them to Gitea immediately
3. **Keep them synchronized** based on your interval
4. **Auto-discover** new repos you create/star on GitHub
5. **Clean up** repos you delete from GitHub
```bash
# Examples of supported formats:
# Option 1: Enable automatic scheduling (triggers auto-start)
SCHEDULE_ENABLED=true
SCHEDULE_INTERVAL=3600 # Check every hour (or use cron: "0 * * * *")
# Option 2: Set mirror interval (also triggers auto-start)
GITEA_MIRROR_INTERVAL=8h # Every 8 hours
GITEA_MIRROR_INTERVAL=30m # Every 30 minutes
GITEA_MIRROR_INTERVAL=1d # Daily
GITEA_MIRROR_INTERVAL=86400 # Every 86400 seconds (24 hours)
# Other examples: 5m, 30m, 1h, 24h, 1d, 7d
# Advanced: Use cron expressions for specific times
SCHEDULE_INTERVAL="0 2 * * *" # Daily at 2 AM (optimize bandwidth usage)
# Auto-import new repositories (default: true)
AUTO_IMPORT_REPOS=true
# Auto-cleanup orphaned repositories
CLEANUP_DELETE_IF_NOT_IN_GITHUB=true
CLEANUP_ORPHANED_REPO_ACTION=archive # 'archive' (recommended) or 'delete'
CLEANUP_DRY_RUN=false # Set to true to test without changes
```
When this variable is set, the scheduler automatically enables and runs at the specified interval. The timer starts from the last successful sync, not from container startup.
**Important Notes**:
- **Auto-Start**: When `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` is set, the service automatically imports all GitHub repositories and mirrors them on startup. No manual "Import" or "Mirror" button clicks required!
- The scheduler checks every minute for tasks to run. The `GITEA_MIRROR_INTERVAL` determines how often each repository is actually synced. For example, with `8h`, each repo syncs every 8 hours from its last successful sync.
**🛡️ Backup Protection Features**:
- **No Accidental Deletions**: Repository cleanup is automatically skipped if GitHub is inaccessible (account deleted, banned, or API errors)
- **Archive Never Deletes Data**: The `archive` action preserves all repository data:
- Regular repositories: Made read-only using Gitea's archive feature
- Mirror repositories: Renamed with `archived-` prefix (Gitea API limitation prevents archiving mirrors)
- Failed operations: Repository remains fully accessible even if marking as archived fails
- **Manual Sync on Demand**: Archived mirrors stay in Gitea with automatic syncs disabled; trigger `Manual Sync` from the Repositories page whenever you need fresh data.
- **The Whole Point of Backups**: Your Gitea mirrors are preserved even when GitHub sources disappear - that's why you have backups!
- **Strongly Recommended**: Always use `CLEANUP_ORPHANED_REPO_ACTION=archive` (default) instead of `delete`
## Troubleshooting

View File

@@ -14,9 +14,9 @@ export default defineConfig({
plugins: [tailwindcss()],
build: {
rollupOptions: {
external: ['bun']
}
}
external: ['bun', 'bun:*'],
},
},
},
integrations: [react()]
});
});

503
bun.lock

File diff suppressed because it is too large Load Diff

View File

@@ -1,5 +1,7 @@
# Gitea Mirror alternate deployment configuration
# Standard deployment with host path and minimal environments
# Minimal Gitea Mirror deployment
# Only includes what CANNOT be configured via the Web UI
# Everything else can be set up through the web interface after deployment
services:
gitea-mirror:
image: ghcr.io/raylabshq/gitea-mirror:latest
@@ -11,17 +13,45 @@ services:
volumes:
- ./data:/app/data
environment:
# For a complete list of all supported environment variables, see:
# docs/ENVIRONMENT_VARIABLES.md or .env.example
# === ABSOLUTELY REQUIRED ===
# This MUST be set and CANNOT be changed via UI
- BETTER_AUTH_SECRET=${BETTER_AUTH_SECRET} # Min 32 chars, required for sessions
- BETTER_AUTH_URL=${BETTER_AUTH_URL:-http://localhost:4321}
- BETTER_AUTH_TRUSTED_ORIGINS=${BETTER_AUTH_TRUSTED_ORIGINS:-http://localhost:4321}
# === CORE SETTINGS ===
# These are technically required but have working defaults
- NODE_ENV=production
- DATABASE_URL=file:data/gitea-mirror.db
- HOST=0.0.0.0
- PORT=4321
- BETTER_AUTH_URL=http://localhost:4321
- BETTER_AUTH_SECRET=${BETTER_AUTH_SECRET:-your-secret-key-change-this-in-production}
- PUBLIC_BETTER_AUTH_URL=${PUBLIC_BETTER_AUTH_URL:-http://localhost:4321}
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=3", "--spider", "http://localhost:4321/api/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 15s
# === QUICK START ===
#
# 1. Create a .env file with only ONE required variable:
# BETTER_AUTH_SECRET=your-32-character-minimum-secret-key-here
#
# 2. Run:
# docker-compose -f docker-compose.alt.yml up -d
#
# 3. Access at http://localhost:4321
#
# 4. Sign up for an account (first user becomes admin)
#
# 5. Configure everything else through the web UI:
# - GitHub credentials
# - Gitea credentials
# - Mirror settings
# - Scheduling options
# - Auto-import settings
# - Cleanup preferences
#
# That's it! Everything else can be configured via the web interface.

View File

@@ -1,174 +0,0 @@
version: "3.8"
services:
# PostgreSQL database for Authentik
authentik-db:
image: postgres:15-alpine
container_name: authentik-db
restart: unless-stopped
environment:
POSTGRES_USER: authentik
POSTGRES_PASSWORD: authentik-db-password
POSTGRES_DB: authentik
volumes:
- authentik-db-data:/var/lib/postgresql/data
networks:
- authentik-net
healthcheck:
test: ["CMD-SHELL", "pg_isready -U authentik"]
interval: 10s
timeout: 5s
retries: 5
# Redis cache for Authentik
authentik-redis:
image: redis:7-alpine
container_name: authentik-redis
restart: unless-stopped
command: redis-server --save 60 1 --loglevel warning
volumes:
- authentik-redis-data:/data
networks:
- authentik-net
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 10s
timeout: 5s
retries: 5
# Authentik Server
authentik-server:
image: ghcr.io/goauthentik/server:2024.2
container_name: authentik-server
restart: unless-stopped
command: server
environment:
# Core Settings
AUTHENTIK_SECRET_KEY: "change-me-to-a-random-50-char-string-for-production"
AUTHENTIK_ERROR_REPORTING__ENABLED: false
# Database
AUTHENTIK_POSTGRESQL__HOST: authentik-db
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: authentik-db-password
# Redis
AUTHENTIK_REDIS__HOST: authentik-redis
# Email (optional - for testing, uses console backend)
AUTHENTIK_EMAIL__HOST: localhost
AUTHENTIK_EMAIL__PORT: 25
AUTHENTIK_EMAIL__USE_TLS: false
AUTHENTIK_EMAIL__USE_SSL: false
AUTHENTIK_EMAIL__TIMEOUT: 10
AUTHENTIK_EMAIL__FROM: authentik@localhost
# Log Level
AUTHENTIK_LOG_LEVEL: info
# Disable analytics
AUTHENTIK_DISABLE_UPDATE_CHECK: true
AUTHENTIK_DISABLE_STARTUP_ANALYTICS: true
# Default admin user (only created on first run)
AUTHENTIK_BOOTSTRAP_PASSWORD: admin-password
AUTHENTIK_BOOTSTRAP_TOKEN: initial-admin-token
AUTHENTIK_BOOTSTRAP_EMAIL: admin@example.com
volumes:
- authentik-media:/media
- authentik-templates:/templates
ports:
- "9000:9000" # HTTP
- "9443:9443" # HTTPS (if configured)
networks:
- authentik-net
- gitea-mirror-net
depends_on:
authentik-db:
condition: service_healthy
authentik-redis:
condition: service_healthy
# Authentik Worker (background tasks)
authentik-worker:
image: ghcr.io/goauthentik/server:2024.2
container_name: authentik-worker
restart: unless-stopped
command: worker
environment:
# Same environment as server
AUTHENTIK_SECRET_KEY: "change-me-to-a-random-50-char-string-for-production"
AUTHENTIK_ERROR_REPORTING__ENABLED: false
AUTHENTIK_POSTGRESQL__HOST: authentik-db
AUTHENTIK_POSTGRESQL__USER: authentik
AUTHENTIK_POSTGRESQL__NAME: authentik
AUTHENTIK_POSTGRESQL__PASSWORD: authentik-db-password
AUTHENTIK_REDIS__HOST: authentik-redis
AUTHENTIK_EMAIL__HOST: localhost
AUTHENTIK_EMAIL__PORT: 25
AUTHENTIK_EMAIL__USE_TLS: false
AUTHENTIK_EMAIL__USE_SSL: false
AUTHENTIK_EMAIL__TIMEOUT: 10
AUTHENTIK_EMAIL__FROM: authentik@localhost
AUTHENTIK_LOG_LEVEL: info
AUTHENTIK_DISABLE_UPDATE_CHECK: true
AUTHENTIK_DISABLE_STARTUP_ANALYTICS: true
volumes:
- authentik-media:/media
- authentik-templates:/templates
networks:
- authentik-net
depends_on:
authentik-db:
condition: service_healthy
authentik-redis:
condition: service_healthy
# Gitea Mirror Application (uncomment to run together)
# gitea-mirror:
# build: .
# # OR use pre-built image:
# # image: ghcr.io/raylabshq/gitea-mirror:latest
# container_name: gitea-mirror
# restart: unless-stopped
# environment:
# # Core Settings
# BETTER_AUTH_URL: http://localhost:4321
# BETTER_AUTH_TRUSTED_ORIGINS: http://localhost:4321,http://localhost:9000
# BETTER_AUTH_SECRET: "your-32-character-secret-key-here"
#
# # GitHub Settings (configure as needed)
# GITHUB_USERNAME: ${GITHUB_USERNAME}
# GITHUB_TOKEN: ${GITHUB_TOKEN}
#
# # Gitea Settings (configure as needed)
# GITEA_URL: ${GITEA_URL}
# GITEA_USERNAME: ${GITEA_USERNAME}
# GITEA_TOKEN: ${GITEA_TOKEN}
# volumes:
# - ./data:/app/data
# ports:
# - "4321:4321"
# networks:
# - gitea-mirror-net
# depends_on:
# - authentik-server
volumes:
authentik-db-data:
name: authentik-db-data
authentik-redis-data:
name: authentik-redis-data
authentik-media:
name: authentik-media
authentik-templates:
name: authentik-templates
networks:
authentik-net:
name: authentik-net
driver: bridge
gitea-mirror-net:
name: gitea-mirror-net
driver: bridge

View File

@@ -1,130 +0,0 @@
version: "3.8"
services:
# PostgreSQL database for Keycloak
keycloak-db:
image: postgres:15-alpine
container_name: keycloak-db
restart: unless-stopped
environment:
POSTGRES_DB: keycloak
POSTGRES_USER: keycloak
POSTGRES_PASSWORD: keycloak-db-password
volumes:
- keycloak-db-data:/var/lib/postgresql/data
networks:
- keycloak-net
healthcheck:
test: ["CMD-SHELL", "pg_isready -U keycloak"]
interval: 10s
timeout: 5s
retries: 5
# Keycloak Identity Provider
keycloak:
image: quay.io/keycloak/keycloak:23.0
container_name: keycloak
restart: unless-stopped
command: start-dev # Use 'start' for production with HTTPS
environment:
# Admin credentials
KEYCLOAK_ADMIN: admin
KEYCLOAK_ADMIN_PASSWORD: admin-password
# Database configuration
KC_DB: postgres
KC_DB_URL_HOST: keycloak-db
KC_DB_URL_DATABASE: keycloak
KC_DB_USERNAME: keycloak
KC_DB_PASSWORD: keycloak-db-password
# HTTP settings
KC_HTTP_ENABLED: true
KC_HTTP_PORT: 8080
KC_HOSTNAME_STRICT: false
KC_HOSTNAME_STRICT_HTTPS: false
KC_PROXY: edge # If behind a proxy
# Development settings (remove for production)
KC_HOSTNAME: localhost
KC_HOSTNAME_PORT: 8080
KC_HOSTNAME_ADMIN: localhost
# Features
KC_FEATURES: token-exchange,admin-fine-grained-authz
# Health and metrics
KC_HEALTH_ENABLED: true
KC_METRICS_ENABLED: true
# Log level
KC_LOG_LEVEL: INFO
# Uncomment for debug logging
# KC_LOG_LEVEL: DEBUG
# QUARKUS_LOG_CATEGORY__ORG_KEYCLOAK_SERVICES: DEBUG
ports:
- "8080:8080" # HTTP
- "8443:8443" # HTTPS (if configured)
- "9000:9000" # Management
networks:
- keycloak-net
- gitea-mirror-net
depends_on:
keycloak-db:
condition: service_healthy
volumes:
# For custom themes (optional)
- keycloak-themes:/opt/keycloak/themes
# For importing realm configurations
- ./keycloak-realm-export.json:/opt/keycloak/data/import/realm.json:ro
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8080/health/ready"]
interval: 15s
timeout: 10s
retries: 10
start_period: 60s
# Gitea Mirror Application (uncomment to run together)
# gitea-mirror:
# build: .
# # OR use pre-built image:
# # image: ghcr.io/raylabshq/gitea-mirror:latest
# container_name: gitea-mirror
# restart: unless-stopped
# environment:
# # Core Settings
# BETTER_AUTH_URL: http://localhost:4321
# BETTER_AUTH_TRUSTED_ORIGINS: http://localhost:4321,http://localhost:8080
# BETTER_AUTH_SECRET: "your-32-character-secret-key-here"
#
# # GitHub Settings (configure as needed)
# GITHUB_USERNAME: ${GITHUB_USERNAME}
# GITHUB_TOKEN: ${GITHUB_TOKEN}
#
# # Gitea Settings (configure as needed)
# GITEA_URL: ${GITEA_URL}
# GITEA_USERNAME: ${GITEA_USERNAME}
# GITEA_TOKEN: ${GITEA_TOKEN}
# volumes:
# - ./data:/app/data
# ports:
# - "4321:4321"
# networks:
# - gitea-mirror-net
# depends_on:
# keycloak:
# condition: service_healthy
volumes:
keycloak-db-data:
name: keycloak-db-data
keycloak-themes:
name: keycloak-themes
networks:
keycloak-net:
name: keycloak-net
driver: bridge
gitea-mirror-net:
name: gitea-mirror-net
driver: bridge

View File

@@ -53,6 +53,15 @@ services:
- GITEA_ORGANIZATION=${GITEA_ORGANIZATION:-github-mirrors}
- GITEA_ORG_VISIBILITY=${GITEA_ORG_VISIBILITY:-public}
- DELAY=${DELAY:-3600}
# Scheduling and Sync Configuration (Issue #72 fixes)
- SCHEDULE_ENABLED=${SCHEDULE_ENABLED:-false}
- GITEA_MIRROR_INTERVAL=${GITEA_MIRROR_INTERVAL:-8h}
- AUTO_IMPORT_REPOS=${AUTO_IMPORT_REPOS:-true}
- AUTO_MIRROR_REPOS=${AUTO_MIRROR_REPOS:-false}
# Repository Cleanup Configuration
- CLEANUP_DELETE_IF_NOT_IN_GITHUB=${CLEANUP_DELETE_IF_NOT_IN_GITHUB:-false}
- CLEANUP_ORPHANED_REPO_ACTION=${CLEANUP_ORPHANED_REPO_ACTION:-archive}
- CLEANUP_DRY_RUN=${CLEANUP_DRY_RUN:-true}
# Optional: Skip TLS verification (insecure, use only for testing)
# - GITEA_SKIP_TLS_VERIFY=${GITEA_SKIP_TLS_VERIFY:-false}
# Header Authentication (for Reverse Proxy SSO)

View File

@@ -120,156 +120,13 @@ fi
# Dependencies are already installed during the Docker build process
# Initialize the database if it doesn't exist
# Note: Drizzle migrations will be run automatically when the app starts (see src/lib/db/index.ts)
if [ ! -f "/app/data/gitea-mirror.db" ]; then
echo "Initializing database..."
if [ -f "dist/scripts/init-db.js" ]; then
bun dist/scripts/init-db.js
elif [ -f "dist/scripts/manage-db.js" ]; then
bun dist/scripts/manage-db.js init
elif [ -f "scripts/manage-db.ts" ]; then
bun scripts/manage-db.ts init
else
echo "Warning: Could not find database initialization scripts in dist/scripts."
echo "Creating and initializing database manually..."
# Create the database file
touch /app/data/gitea-mirror.db
# Initialize the database with required tables
sqlite3 /app/data/gitea-mirror.db <<EOF
CREATE TABLE IF NOT EXISTS users (
id TEXT PRIMARY KEY,
username TEXT NOT NULL,
password TEXT NOT NULL,
email TEXT NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS configs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
github_config TEXT NOT NULL,
gitea_config TEXT NOT NULL,
include TEXT NOT NULL DEFAULT '["*"]',
exclude TEXT NOT NULL DEFAULT '[]',
schedule_config TEXT NOT NULL,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id)
);
CREATE TABLE IF NOT EXISTS repositories (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
full_name TEXT NOT NULL,
url TEXT NOT NULL,
clone_url TEXT NOT NULL,
owner TEXT NOT NULL,
organization TEXT,
mirrored_location TEXT DEFAULT '',
is_private INTEGER NOT NULL DEFAULT 0,
is_fork INTEGER NOT NULL DEFAULT 0,
forked_from TEXT,
has_issues INTEGER NOT NULL DEFAULT 0,
is_starred INTEGER NOT NULL DEFAULT 0,
is_archived INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
has_lfs INTEGER NOT NULL DEFAULT 0,
has_submodules INTEGER NOT NULL DEFAULT 0,
default_branch TEXT NOT NULL,
visibility TEXT NOT NULL DEFAULT 'public',
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
);
CREATE TABLE IF NOT EXISTS organizations (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT NOT NULL,
membership_role TEXT NOT NULL DEFAULT 'member',
is_included INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
repository_count INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
);
CREATE TABLE IF NOT EXISTS mirror_jobs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
repository_id TEXT,
repository_name TEXT,
organization_id TEXT,
organization_name TEXT,
details TEXT,
status TEXT NOT NULL DEFAULT 'imported',
message TEXT NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- New fields for job resilience
job_type TEXT NOT NULL DEFAULT 'mirror',
batch_id TEXT,
total_items INTEGER,
completed_items INTEGER DEFAULT 0,
item_ids TEXT, -- JSON array as text
completed_item_ids TEXT DEFAULT '[]', -- JSON array as text
in_progress INTEGER NOT NULL DEFAULT 0, -- Boolean as integer
started_at TIMESTAMP,
completed_at TIMESTAMP,
last_checkpoint TIMESTAMP,
FOREIGN KEY (user_id) REFERENCES users(id)
);
CREATE INDEX IF NOT EXISTS idx_mirror_jobs_user_id ON mirror_jobs(user_id);
CREATE INDEX IF NOT EXISTS idx_mirror_jobs_batch_id ON mirror_jobs(batch_id);
CREATE INDEX IF NOT EXISTS idx_mirror_jobs_in_progress ON mirror_jobs(in_progress);
CREATE INDEX IF NOT EXISTS idx_mirror_jobs_job_type ON mirror_jobs(job_type);
CREATE INDEX IF NOT EXISTS idx_mirror_jobs_timestamp ON mirror_jobs(timestamp);
CREATE TABLE IF NOT EXISTS events (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
channel TEXT NOT NULL,
payload TEXT NOT NULL,
read INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id)
);
CREATE INDEX IF NOT EXISTS idx_events_user_channel ON events(user_id, channel);
CREATE INDEX IF NOT EXISTS idx_events_created_at ON events(created_at);
CREATE INDEX IF NOT EXISTS idx_events_read ON events(read);
EOF
echo "Database initialized with required tables."
fi
echo "Database not found. It will be created and initialized via Drizzle migrations on first app startup..."
# Create empty database file so migrations can run
touch /app/data/gitea-mirror.db
else
echo "Database already exists, checking for issues..."
if [ -f "dist/scripts/fix-db-issues.js" ]; then
bun dist/scripts/fix-db-issues.js
elif [ -f "dist/scripts/manage-db.js" ]; then
bun dist/scripts/manage-db.js fix
elif [ -f "scripts/manage-db.ts" ]; then
bun scripts/manage-db.ts fix
fi
echo "Database exists, checking integrity..."
echo "Database already exists, Drizzle will check for pending migrations on startup..."
fi
# Extract version from package.json and set as environment variable

View File

@@ -36,6 +36,7 @@ Essential application settings required for running Gitea Mirror.
| `DATABASE_URL` | Database connection URL | `sqlite://data/gitea-mirror.db` | No |
| `BETTER_AUTH_SECRET` | Secret key for session signing (generate with: `openssl rand -base64 32`) | - | Yes |
| `BETTER_AUTH_URL` | Primary base URL for authentication. This should be the main URL where your application is accessed. | `http://localhost:4321` | No |
| `PUBLIC_BETTER_AUTH_URL` | Client-side auth URL for multi-origin access. Set this to your primary domain when you need to access the app from different origins (e.g., both IP and domain). The client will use this URL for all auth requests instead of the current browser origin. | - | No |
| `BETTER_AUTH_TRUSTED_ORIGINS` | Trusted origins for authentication requests. Comma-separated list of URLs. Use this to specify additional access URLs (e.g., local IP + domain: `http://10.10.20.45:4321,https://gitea-mirror.mydomain.tld`), SSO providers, reverse proxies, etc. | - | No |
| `ENCRYPTION_SECRET` | Optional encryption key for tokens (generate with: `openssl rand -base64 48`) | - | No |
@@ -133,6 +134,7 @@ Control what content gets mirrored from GitHub to Gitea.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `MIRROR_RELEASES` | Mirror GitHub releases | `false` | `true`, `false` |
| `RELEASE_LIMIT` | Maximum number of releases to mirror per repository | `10` | Number (1-100) |
| `MIRROR_WIKI` | Mirror wiki content | `false` | `true`, `false` |
| `MIRROR_METADATA` | Master toggle for metadata mirroring | `false` | `true`, `false` |
| `MIRROR_ISSUES` | Mirror issues (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
@@ -148,10 +150,29 @@ Configure automatic scheduled mirroring.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `SCHEDULE_ENABLED` | Enable automatic mirroring | `false` | `true`, `false` |
| `SCHEDULE_INTERVAL` | Interval in seconds or cron expression | `3600` | Number or cron string (e.g., `"0 2 * * *"`) |
| `SCHEDULE_ENABLED` | Enable automatic mirroring. **When set to `true`, automatically imports and mirrors all repositories on startup** (v3.5.3+) | `false` | `true`, `false` |
| `SCHEDULE_INTERVAL` | Interval in seconds or cron expression. **Supports cron syntax for scheduled runs** (e.g., `"0 2 * * *"` for 2 AM daily) | `3600` | Number (seconds) or cron string |
| `DELAY` | Legacy: same as SCHEDULE_INTERVAL | `3600` | Number (seconds) |
> **🚀 Auto-Start Feature (v3.5.3+)**
> Setting either `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` triggers auto-start functionality where the service will:
> 1. **Import** all GitHub repositories on startup
> 2. **Mirror** them to Gitea immediately
> 3. **Continue syncing** at the configured interval
> 4. **Auto-discover** new repositories
> 5. **Clean up** deleted repositories (if configured)
>
> This eliminates the need for manual button clicks - perfect for Docker/Kubernetes deployments!
> **⏰ Scheduling with Cron Expressions**
> Use cron expressions in `SCHEDULE_INTERVAL` to run at specific times:
> - `"0 2 * * *"` - Daily at 2 AM
> - `"0 */6 * * *"` - Every 6 hours
> - `"0 0 * * 0"` - Weekly on Sunday at midnight
> - `"0 3 * * 1-5"` - Weekdays at 3 AM (Monday-Friday)
>
> This is useful for optimizing bandwidth usage during low-activity periods.
### Execution Settings
| Variable | Description | Default | Options |
@@ -173,6 +194,8 @@ Configure automatic scheduled mirroring.
| Variable | Description | Default | Options |
|----------|-------------|---------|---------|
| `AUTO_IMPORT_REPOS` | Automatically discover and import new GitHub repositories during scheduled syncs | `true` | `true`, `false` |
| `AUTO_MIRROR_REPOS` | Automatically mirror newly imported repositories during scheduled syncs (no manual “Mirror All” required) | `false` | `true`, `false` |
| `SCHEDULE_ONLY_MIRROR_UPDATED` | Only mirror repos with updates | `false` | `true`, `false` |
| `SCHEDULE_UPDATE_INTERVAL` | Check for updates interval (milliseconds) | `86400000` | Number |
| `SCHEDULE_SKIP_RECENTLY_MIRRORED` | Skip recently mirrored repos | `true` | `true`, `false` |
@@ -205,10 +228,26 @@ Configure automatic cleanup of old events and data.
|----------|-------------|---------|---------|
| `CLEANUP_DELETE_FROM_GITEA` | Delete repositories from Gitea | `false` | `true`, `false` |
| `CLEANUP_DELETE_IF_NOT_IN_GITHUB` | Delete repos not found in GitHub (automatically enables cleanup) | `true` | `true`, `false` |
| `CLEANUP_ORPHANED_REPO_ACTION` | Action for orphaned repositories | `archive` | `skip`, `archive`, `delete` |
| `CLEANUP_DRY_RUN` | Test mode without actual deletion | `true` | `true`, `false` |
| `CLEANUP_ORPHANED_REPO_ACTION` | Action for orphaned repositories. **Note**: `archive` is recommended to preserve backups | `archive` | `skip`, `archive`, `delete` |
| `CLEANUP_DRY_RUN` | Test mode without actual deletion | `false` | `true`, `false` |
| `CLEANUP_PROTECTED_REPOS` | Comma-separated list of protected repository names | - | Comma-separated strings |
**🛡️ Safety Features (Backup Protection)**:
- **GitHub Failures Don't Delete Backups**: Cleanup is automatically skipped if GitHub API returns errors (404, 403, connection issues)
- **Archive Never Deletes**: The `archive` action ALWAYS preserves repository data, it never deletes
- **Graceful Degradation**: If marking as archived fails, the repository remains fully accessible in Gitea
- **The Purpose of Backups**: Your mirrors are preserved even when GitHub sources disappear - that's the whole point!
**Archive Behavior (Aligned with Gitea API)**:
- **Regular repositories**: Uses Gitea's native archive feature (PATCH `/repos/{owner}/{repo}` with `archived: true`)
- Makes repository read-only while preserving all data
- **Mirror repositories**: Uses rename strategy (Gitea API returns 422 for archiving mirrors)
- Renamed with `archived-` prefix for clear identification
- Description updated with preservation notice and timestamp
- Mirror interval set to 8760h (1 year) to minimize sync attempts
- Repository remains fully accessible and cloneable
- **Manual Sync Option**: Archived mirrors are still available on the Repositories page with automatic syncs disabled—use the `Manual Sync` action to refresh them on demand.
### Execution Settings
| Variable | Description | Default | Options |
@@ -300,21 +339,28 @@ services:
### Multiple Access URLs
To allow access to Gitea Mirror through multiple URLs (e.g., local IP and public domain), use the `BETTER_AUTH_TRUSTED_ORIGINS` variable:
To allow access to Gitea Mirror through multiple URLs (e.g., local IP and public domain), you need to configure both server and client settings:
**Example Configuration:**
```bash
# Primary URL (required) - typically your public domain
# Primary URL (required) - where the auth server is hosted
BETTER_AUTH_URL=https://gitea-mirror.mydomain.tld
# Additional access URLs (optional) - local IPs, alternate domains
# Client-side URL (optional) - tells the browser where to send auth requests
# Set this to your primary domain when accessing from different origins
PUBLIC_BETTER_AUTH_URL=https://gitea-mirror.mydomain.tld
# Additional trusted origins (optional) - origins allowed to make auth requests
BETTER_AUTH_TRUSTED_ORIGINS=http://10.10.20.45:4321,http://192.168.1.100:4321
```
This setup allows you to:
- Access via local network IP: `http://10.10.20.45:4321`
- Access via public domain: `https://gitea-mirror.mydomain.tld`
- Both URLs will work for authentication and session management
- Auth requests from the IP will be sent to the domain (via `PUBLIC_BETTER_AUTH_URL`)
- Each origin requires separate login due to browser cookie isolation
**Important:** When accessing from different origins (IP vs domain), you'll need to log in separately on each origin as cookies cannot be shared across different origins for security reasons.
### Trusted Origins
@@ -363,4 +409,4 @@ BETTER_AUTH_TRUSTED_ORIGINS=http://localhost:3000,http://192.168.1.100:3000
- `admin:org` (read organization data)
- Additional scopes may be required for specific features
For more examples and detailed configuration, see the `.env.example` file in the repository.
For more examples and detailed configuration, see the `.env.example` file in the repository.

View File

@@ -60,7 +60,7 @@ bun run dev
## Key Features
- 🔄 **Automatic Mirroring** - Keep repositories synchronized
- 🔄 **Automatic Syncing** - Keep repositories synchronized
- 🗂️ **Organization Support** - Mirror entire organizations
-**Starred Repos** - Mirror your starred repositories
- 🔐 **Self-Hosted** - Full control over your data

View File

@@ -0,0 +1,3 @@
ALTER TABLE `organizations` ADD `public_repository_count` integer;--> statement-breakpoint
ALTER TABLE `organizations` ADD `private_repository_count` integer;--> statement-breakpoint
ALTER TABLE `organizations` ADD `fork_repository_count` integer;

View File

@@ -0,0 +1,18 @@
CREATE TABLE `rate_limits` (
`id` text PRIMARY KEY NOT NULL,
`user_id` text NOT NULL,
`provider` text DEFAULT 'github' NOT NULL,
`limit` integer NOT NULL,
`remaining` integer NOT NULL,
`used` integer NOT NULL,
`reset` integer NOT NULL,
`retry_after` integer,
`status` text DEFAULT 'ok' NOT NULL,
`last_checked` integer NOT NULL,
`created_at` integer DEFAULT (unixepoch()) NOT NULL,
`updated_at` integer DEFAULT (unixepoch()) NOT NULL,
FOREIGN KEY (`user_id`) REFERENCES `users`(`id`) ON UPDATE no action ON DELETE no action
);
--> statement-breakpoint
CREATE INDEX `idx_rate_limits_user_provider` ON `rate_limits` (`user_id`,`provider`);--> statement-breakpoint
CREATE INDEX `idx_rate_limits_status` ON `rate_limits` (`status`);

View File

@@ -0,0 +1,11 @@
-- Step 1: Remove duplicate repositories, keeping the most recently updated one
-- This handles cases where users have duplicate entries from before the unique constraint
DELETE FROM repositories
WHERE rowid NOT IN (
SELECT MAX(rowid)
FROM repositories
GROUP BY user_id, full_name
);
--> statement-breakpoint
-- Step 2: Now create the unique index safely
CREATE UNIQUE INDEX uniq_repositories_user_full_name ON repositories (user_id, full_name);

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -22,6 +22,27 @@
"when": 1753539600567,
"tag": "0002_bored_captain_cross",
"breakpoints": true
},
{
"idx": 3,
"version": "6",
"when": 1757390828679,
"tag": "0003_open_spacker_dave",
"breakpoints": true
},
{
"idx": 4,
"version": "6",
"when": 1757392620734,
"tag": "0004_grey_butterfly",
"breakpoints": true
},
{
"idx": 5,
"version": "6",
"when": 1757786449446,
"tag": "0005_polite_preak",
"breakpoints": true
}
]
}

View File

@@ -0,0 +1,21 @@
---
extends: default
ignore: |
.yamllint
node_modules
templates
unittests/bash
rules:
truthy:
allowed-values: ['true', 'false']
check-keys: False
level: error
line-length: disable
document-start: disable
comments:
min-spaces-from-content: 1
braces:
max-spaces-inside: 2

View File

@@ -0,0 +1,12 @@
apiVersion: v2
name: gitea-mirror
description: Kubernetes helm chart for gitea-mirror
type: application
version: 0.0.1
appVersion: 3.7.2
icon: https://github.com/RayLabsHQ/gitea-mirror/blob/main/.github/assets/logo.png
keywords:
- git
- gitea
sources:
- https://github.com/RayLabsHQ/gitea-mirror

307
helm/gitea-mirror/README.md Normal file
View File

@@ -0,0 +1,307 @@
# gitea-mirror (Helm Chart)
Deploy **gitea-mirror** to Kubernetes using Helm. The chart packages a Deployment, Service, optional Ingress or Gateway API HTTPRoutes, ConfigMap and Secret, a PVC (optional), and an optional ServiceAccount.
- **Chart name:** `gitea-mirror`
- **Type:** `application`
- **App version:** `3.7.2` (default image tag, can be overridden)
---
## Prerequisites
- Kubernetes 1.23+
- Helm 3.8+
- (Optional) Gateway API (v1) if you plan to use `route.*` HTTPRoutes, see https://github.com/kubernetes-sigs/gateway-api/
- (Optional) An Ingress controller if you plan to use `ingress.*`
---
## Quick start
From the repo root (chart path: `helm/gitea-mirror`):
```bash
# Create a namespace (optional)
kubectl create namespace gitea-mirror
# Install with minimal required secrets/values
helm upgrade --install gitea-mirror ./helm/gitea-mirror --namespace gitea-mirror --set "gitea-mirror.github.username=<your-gh-username>" --set "gitea-mirror.github.token=<your-gh-token>" --set "gitea-mirror.gitea.url=https://gitea.example.com" --set "gitea-mirror.gitea.token=<your-gitea-token>"
```
The default Service is `ClusterIP` on port `8080`. You can expose it via Ingress or Gateway API; see below.
---
## Upgrading
Standard Helm upgrade:
```bash
helm upgrade gitea-mirror ./helm/gitea-mirror -n gitea-mirror
```
If you change persistence settings or storage class, a rollout may require PVC recreation.
---
## Uninstalling
```bash
helm uninstall gitea-mirror -n gitea-mirror
```
If you enabled persistence with a PVC the data may persist; delete the PVC manually if you want a clean slate.
---
## Configuration
### Global image & pod settings
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `image.registry` | string | `ghcr.io` | Container registry. |
| `image.repository` | string | `raylabshq/gitea-mirror` | Image repository. |
| `image.tag` | string | `""` | Image tag; when empty, uses the chart `appVersion` (`3.7.2`). |
| `image.pullPolicy` | string | `IfNotPresent` | K8s image pull policy. |
| `imagePullSecrets` | list | `[]` | Image pull secrets. |
| `podSecurityContext.runAsUser` | int | `1001` | UID. |
| `podSecurityContext.runAsGroup` | int | `1001` | GID. |
| `podSecurityContext.fsGroup` | int | `1001` | FS group. |
| `podSecurityContext.fsGroupChangePolicy` | string | `OnRootMismatch` | FS group change policy. |
| `nodeSelector` / `tolerations` / `affinity` / `topologySpreadConstraints` | — | — | Standard scheduling knobs. |
| `extraVolumes` / `extraVolumeMounts` | list | `[]` | Append custom volumes/mounts. |
| `priorityClassName` | string | `""` | Optional Pod priority class. |
### Deployment
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `deployment.port` | int | `8080` | Container port & named `http` port. |
| `deployment.strategy.type` | string | `Recreate` | Update strategy (`Recreate` or `RollingUpdate`). |
| `deployment.strategy.rollingUpdate.maxUnavailable/maxSurge` | string/int | — | Used when `type=RollingUpdate`. |
| `deployment.env` | list | `[]` | Extra environment variables. |
| `deployment.resources` | map | `{}` | CPU/memory requests & limits. |
| `deployment.terminationGracePeriodSeconds` | int | `60` | Grace period. |
| `livenessProbe.*` | — | enabled, `/api/health` | Liveness probe (HTTP GET to `/api/health`). |
| `readinessProbe.*` | — | enabled, `/api/health` | Readiness probe. |
| `startupProbe.*` | — | enabled, `/api/health` | Startup probe. |
> The Pod mounts a volume at `/app/data` (PVC or `emptyDir` depending on `persistence.enabled`).
### Service
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `service.type` | string | `ClusterIP` | Service type. |
| `service.port` | int | `8080` | Service port. |
| `service.clusterIP` | string | `None` | ClusterIP (only when `type=ClusterIP`). |
| `service.externalTrafficPolicy` | string | `""` | External traffic policy (LB). |
| `service.loadBalancerIP` | string | `""` | LoadBalancer IP. |
| `service.loadBalancerClass` | string | `""` | LoadBalancer class. |
| `service.annotations` / `service.labels` | map | `{}` | Extra metadata. |
### Ingress (optional)
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `ingress.enabled` | bool | `false` | Enable Ingress. |
| `ingress.className` | string | `""` | IngressClass name. |
| `ingress.hosts[0].host` | string | `mirror.example.com` | Hostname. |
| `ingress.tls` | list | `[]` | TLS blocks (secret name etc.). |
| `ingress.annotations` | map | `{}` | Controller-specific annotations. |
> The Ingress exposes `/` to the charts Service.
### Gateway API HTTPRoutes (optional)
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `route.enabled` | bool | `false` | Enable Gateway API HTTPRoutes. |
| `route.forceHTTPS` | bool | `true` | If true, create an HTTP route that redirects to HTTPS (301). |
| `route.domain` | list | `["mirror.example.com"]` | Hostnames. |
| `route.gateway` | string | `""` | Gateway name. |
| `route.gatewayNamespace` | string | `""` | Gateway namespace. |
| `route.http.gatewaySection` | string | `""` | SectionName for HTTP listener. |
| `route.https.gatewaySection` | string | `""` | SectionName for HTTPS listener. |
| `route.http.filters` / `route.https.filters` | list | `[]` | Additional filters. (Defaults add HSTS header on HTTPS.) |
### Persistence
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `persistence.enabled` | bool | `true` | Enable persistent storage. |
| `persistence.create` | bool | `true` | Create a PVC from the chart. |
| `persistence.claimName` | string | `gitea-mirror-storage` | PVC name. |
| `persistence.storageClass` | string | `""` | StorageClass to use. |
| `persistence.accessModes` | list | `["ReadWriteOnce"]` | Access modes. |
| `persistence.size` | string | `1Gi` | Requested size. |
| `persistence.volumeName` | string | `""` | Bind to existing PV by name (optional). |
| `persistence.annotations` | map | `{}` | PVC annotations. |
### ServiceAccount (optional)
| Key | Type | Default | Description |
| --- | --- | --- | --- |
| `serviceAccount.create` | bool | `false` | Create a ServiceAccount. |
| `serviceAccount.name` | string | `""` | SA name (defaults to release fullname). |
| `serviceAccount.automountServiceAccountToken` | bool | `false` | Automount token. |
| `serviceAccount.annotations` / `labels` | map | `{}` | Extra metadata. |
---
## Application configuration (`gitea-mirror.*`)
These values populate a **ConfigMap** (non-secret) and a **Secret** (for tokens and sensitive fields). Environment variables from both are consumed by the container.
### Core
| Key | Default | Mapped env |
| --- | --- | --- |
| `gitea-mirror.nodeEnv` | `production` | `NODE_ENV` |
| `gitea-mirror.core.databaseUrl` | `file:data/gitea-mirror.db` | `DATABASE_URL` |
| `gitea-mirror.core.encryptionSecret` | `""` | `ENCRYPTION_SECRET` (Secret) |
| `gitea-mirror.core.betterAuthSecret` | `""` | `BETTER_AUTH_SECRET` |
| `gitea-mirror.core.betterAuthUrl` | `http://localhost:4321` | `BETTER_AUTH_URL` |
| `gitea-mirror.core.betterAuthTrustedOrigins` | `http://localhost:4321` | `BETTER_AUTH_TRUSTED_ORIGINS` |
### GitHub
| Key | Default | Mapped env |
| --- | --- | --- |
| `gitea-mirror.github.username` | `""` | `GITHUB_USERNAME` |
| `gitea-mirror.github.token` | `""` | `GITHUB_TOKEN` (Secret) |
| `gitea-mirror.github.type` | `personal` | `GITHUB_TYPE` |
| `gitea-mirror.github.privateRepositories` | `true` | `PRIVATE_REPOSITORIES` |
| `gitea-mirror.github.skipForks` | `false` | `SKIP_FORKS` |
| `gitea-mirror.github.starredCodeOnly` | `false` | `SKIP_STARRED_ISSUES` |
| `gitea-mirror.github.mirrorStarred` | `false` | `MIRROR_STARRED` |
### Gitea
| Key | Default | Mapped env |
| --- | --- | --- |
| `gitea-mirror.gitea.url` | `""` | `GITEA_URL` |
| `gitea-mirror.gitea.token` | `""` | `GITEA_TOKEN` (Secret) |
| `gitea-mirror.gitea.username` | `""` | `GITEA_USERNAME` |
| `gitea-mirror.gitea.organization` | `github-mirrors` | `GITEA_ORGANIZATION` |
| `gitea-mirror.gitea.visibility` | `public` | `GITEA_ORG_VISIBILITY` |
### Mirror options
| Key | Default | Mapped env |
| --- | --- | --- |
| `gitea-mirror.mirror.releases` | `true` | `MIRROR_RELEASES` |
| `gitea-mirror.mirror.wiki` | `true` | `MIRROR_WIKI` |
| `gitea-mirror.mirror.metadata` | `true` | `MIRROR_METADATA` |
| `gitea-mirror.mirror.issues` | `true` | `MIRROR_ISSUES` |
| `gitea-mirror.mirror.pullRequests` | `true` | `MIRROR_PULL_REQUESTS` |
| `gitea-mirror.mirror.starred` | _(see note above)_ | `MIRROR_STARRED` |
### Automation & cleanup
| Key | Default | Mapped env |
| --- | --- | --- |
| `gitea-mirror.automation.schedule_enabled` | `true` | `SCHEDULE_ENABLED` |
| `gitea-mirror.automation.schedule_interval` | `3600` | `SCHEDULE_INTERVAL` (seconds) |
| `gitea-mirror.cleanup.enabled` | `true` | `CLEANUP_ENABLED` |
| `gitea-mirror.cleanup.retentionDays` | `30` | `CLEANUP_RETENTION_DAYS` |
> **Secrets:** If you set `gitea-mirror.existingSecret` (name of an existing Secret), the chart will **not** create its own Secret and will reference yours instead. Otherwise it creates a Secret with `GITHUB_TOKEN`, `GITEA_TOKEN`, `ENCRYPTION_SECRET`.
---
## Exposing the service
### Using Ingress
```yaml
ingress:
enabled: true
className: "nginx"
hosts:
- host: mirror.example.com
tls:
- secretName: mirror-tls
hosts:
- mirror.example.com
```
This creates an Ingress routing `/` to the service on port `8080`.
### Using Gateway API (HTTPRoute)
```yaml
route:
enabled: true
domain: ["mirror.example.com"]
gateway: "my-gateway"
gatewayNamespace: "gateway-system"
http:
gatewaySection: "http"
https:
gatewaySection: "https"
# Example extra filter already included by default: add HSTS header
```
If `forceHTTPS: true`, the chart emits an HTTP route that redirects to HTTPS with 301. An HTTPS route is always created when `route.enabled=true`.
---
## Persistence & data
By default, the chart provisions a PVC named `gitea-mirror-storage` with `1Gi` and mounts it at `/app/data`. To use an existing PV or tune storage, adjust `persistence.*` in `values.yaml`. If you disable persistence, an `emptyDir` will be used instead.
---
## Environment & health endpoints
The container listens on `PORT` (defaults to `deployment.port` = `8080`) and exposes `GET /api/health` for liveness/readiness/startup probes.
---
## Examples
### Minimal (tokens via chart-managed Secret)
```yaml
gitea-mirror:
github:
username: "gitea-mirror"
token: "<gh-token>"
gitea:
url: "https://gitea.company.tld"
token: "<gitea-token>"
```
### Bring your own Secret
```yaml
gitea-mirror:
existingSecret: "gitea-mirror-secrets"
github:
username: "gitea-mirror"
gitea:
url: "https://gitea.company.tld"
```
Where `gitea-mirror-secrets` contains keys `GITHUB_TOKEN`, `GITEA_TOKEN`, `ENCRYPTION_SECRET`.
---
## Development
Lint the chart:
```bash
yamllint -c helm/gitea-mirror/.yamllint helm/gitea-mirror
```
Tweak probes, resources, and scheduling as needed; see `values.yaml`.
---
## License
This chart is part of the `RayLabsHQ/gitea-mirror` repository. See the repository for licensing details.

View File

@@ -0,0 +1,59 @@
{{/*
Expand the name of the chart.
*/}}
{{- define "gitea-mirror.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "gitea-mirror.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "gitea-mirror.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Common labels
*/}}
{{- define "gitea-mirror.labels" -}}
helm.sh/chart: {{ include "gitea-mirror.chart" . }}
app: {{ include "gitea-mirror.name" . }}
{{ include "gitea-mirror.selectorLabels" . }}
app.kubernetes.io/version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
version: {{ .Values.image.tag | default .Chart.AppVersion | quote }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end -}}
{{/*
Selector labels
*/}}
{{- define "gitea-mirror.selectorLabels" -}}
app.kubernetes.io/name: {{ include "gitea-mirror.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end -}}
{{/*
ServiceAccount name
*/}}
{{- define "gitea-mirror.serviceAccountName" -}}
{{ .Values.serviceAccount.name | default (include "gitea-mirror.fullname" .) }}
{{- end -}}

View File

@@ -0,0 +1,38 @@
{{- $gm := index .Values "gitea-mirror" -}}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ include "gitea-mirror.fullname" . }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
data:
NODE_ENV: {{ $gm.nodeEnv | quote }}
# Core configuration
DATABASE_URL: {{ $gm.core.databaseUrl | quote }}
BETTER_AUTH_SECRET: {{ $gm.core.betterAuthSecret | quote }}
BETTER_AUTH_URL: {{ $gm.core.betterAuthUrl | quote }}
BETTER_AUTH_TRUSTED_ORIGINS: {{ $gm.core.betterAuthTrustedOrigins | quote }}
# GitHub Config
GITHUB_USERNAME: {{ $gm.github.username | quote }}
GITHUB_TYPE: {{ $gm.github.type | quote }}
PRIVATE_REPOSITORIES: {{ $gm.github.privateRepositories | quote }}
MIRROR_STARRED: {{ $gm.github.mirrorStarred | quote }}
SKIP_FORKS: {{ $gm.github.skipForks | quote }}
SKIP_STARRED_ISSUES: {{ $gm.github.starredCodeOnly | quote }}
# Gitea Config
GITEA_URL: {{ $gm.gitea.url | quote }}
GITEA_USERNAME: {{ $gm.gitea.username | quote }}
GITEA_ORGANIZATION: {{ $gm.gitea.organization | quote }}
GITEA_ORG_VISIBILITY: {{ $gm.gitea.visibility | quote }}
# Mirror Options
MIRROR_RELEASES: {{ $gm.mirror.releases | quote }}
MIRROR_WIKI: {{ $gm.mirror.wiki | quote }}
MIRROR_METADATA: {{ $gm.mirror.metadata | quote }}
MIRROR_ISSUES: {{ $gm.mirror.issues | quote }}
MIRROR_PULL_REQUESTS: {{ $gm.mirror.pullRequests | quote }}
# Automation
SCHEDULE_ENABLED: {{ $gm.automation.schedule_enabled| quote }}
SCHEDULE_INTERVAL: {{ $gm.automation.schedule_interval | quote }}
# Cleanup
CLEANUP_ENABLED: {{ $gm.cleanup.enabled | quote }}
CLEANUP_RETENTION_DAYS: {{ $gm.cleanup.retentionDays | quote }}

View File

@@ -0,0 +1,143 @@
{{- $gm := index .Values "gitea-mirror" -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "gitea-mirror.fullname" . }}
{{- with .Values.deployment.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
{{- if .Values.deployment.labels }}
{{- toYaml .Values.deployment.labels | nindent 4 }}
{{- end }}
spec:
replicas: 1
strategy:
type: {{ .Values.deployment.strategy.type }}
{{- if eq .Values.deployment.strategy.type "RollingUpdate" }}
rollingUpdate:
maxUnavailable: {{ .Values.deployment.strategy.rollingUpdate.maxUnavailable }}
maxSurge: {{ .Values.deployment.strategy.rollingUpdate.maxSurge }}
{{- end }}
selector:
matchLabels:
{{- include "gitea-mirror.selectorLabels" . | nindent 6 }}
template:
metadata:
labels:
{{- include "gitea-mirror.labels" . | nindent 8 }}
{{- if .Values.deployment.labels }}
{{- toYaml .Values.deployment.labels | nindent 8 }}
{{- end }}
spec:
{{- if (or .Values.serviceAccount.create .Values.serviceAccount.name) }}
serviceAccountName: {{ include "gitea-mirror.serviceAccountName" . }}
{{- end }}
{{- if .Values.priorityClassName }}
priorityClassName: "{{ .Values.priorityClassName }}"
{{- end }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
terminationGracePeriodSeconds: {{ .Values.deployment.terminationGracePeriodSeconds }}
containers:
- name: gitea-mirror
image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag | default (printf "v%s" .Chart.AppVersion) }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
envFrom:
- configMapRef:
name: {{ include "gitea-mirror.fullname" . }}
{{- if $gm.existingSecret }}
- secretRef:
name: {{ $gm.existingSecret }}
{{- else }}
- secretRef:
name: {{ include "gitea-mirror.fullname" . }}
{{- end }}
env:
- name: PORT
value: "{{ .Values.deployment.port }}"
{{- if .Values.deployment.env }}
{{- toYaml .Values.deployment.env | nindent 12 }}
{{- end }}
ports:
- name: http
containerPort: {{ .Values.deployment.port }}
{{- if .Values.livenessProbe.enabled }}
livenessProbe:
httpGet:
path: /api/health
port: "http"
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
{{- end }}
{{- if .Values.readinessProbe.enabled }}
readinessProbe:
httpGet:
path: /api/health
port: "http"
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
{{- end }}
{{- if .Values.startupProbe.enabled }}
startupProbe:
httpGet:
path: /api/health
port: "http"
initialDelaySeconds: {{ .Values.startupProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.startupProbe.periodSeconds }}
timeoutSeconds: {{ .Values.startupProbe.timeoutSeconds }}
failureThreshold: {{ .Values.startupProbe.failureThreshold }}
successThreshold: {{ .Values.startupProbe.successThreshold }}
{{- end }}
volumeMounts:
- name: data
mountPath: /app/data
{{- if .Values.extraVolumeMounts }}
{{- toYaml .Values.extraVolumeMounts | nindent 12 }}
{{- end }}
{{- with .Values.deployment.resources }}
resources:
{{- toYaml .Values.deployment.resources | nindent 12 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.topologySpreadConstraints }}
topologySpreadConstraints:
{{- toYaml . | nindent 8 }}
{{- end }}
volumes:
{{- if .Values.persistence.enabled }}
- name: data
persistentVolumeClaim:
claimName: {{ .Values.persistence.claimName }}
{{- else if not .Values.persistence.enabled }}
- name: data
emptyDir: {}
{{- end }}
{{- if .Values.extraVolumes }}
{{- toYaml .Values.extraVolumes | nindent 8 }}
{{- end }}

View File

@@ -0,0 +1,77 @@
{{- if .Values.route.enabled }}
{{- if .Values.route.forceHTTPS }}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: {{ include "gitea-mirror.fullname" . }}-http
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
spec:
parentRefs:
- name: {{ .Values.route.gateway }}
sectionName: {{ .Values.route.http.gatewaySection }}
namespace: {{ .Values.route.gatewayNamespace }}
hostnames: {{ .Values.route.domain }}
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
{{- with .Values.route.http.filters }}
{{ toYaml . | nindent 4 }}
{{- end }}
{{- else }}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: {{ include "gitea-mirror.fullname" . }}-http
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
spec:
parentRefs:
- name: {{ .Values.route.gateway }}
sectionName: {{ .Values.route.http.gatewaySection }}
namespace: {{ .Values.route.gatewayNamespace }}
hostnames: {{ .Values.route.domain }}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: {{ include "gitea-mirror.fullname" . }}
port: {{ .Values.service.port }}
{{- with .Values.route.http.filters }}
filters:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: {{ include "gitea-mirror.fullname" . }}-https
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
spec:
parentRefs:
- name: {{ .Values.route.gateway }}
sectionName: {{ .Values.route.https.gatewaySection }}
namespace: {{ .Values.route.gatewayNamespace }}
hostnames: {{ .Values.route.domain }}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: {{ include "gitea-mirror.fullname" . }}
port: {{ .Values.service.port }}
{{- with .Values.route.https.filters }}
filters:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}

View File

@@ -0,0 +1,40 @@
{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: {{ include "gitea-mirror.fullname" . }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- . | toYaml | nindent 4 }}
{{- end }}
spec:
{{- if .Values.ingress.className }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ tpl . $ | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ tpl .host $ | quote }}
http:
paths:
- path: {{ .path | default "/" }}
pathType: {{ .pathType | default "Prefix" }}
backend:
service:
name: {{ include "gitea-mirror.fullname" $ }}
port:
number: {{ $.Values.service.port }}
{{- end }}
{{- end }}

View File

@@ -0,0 +1,26 @@
{{- if and .Values.persistence.enabled .Values.persistence.create }}
{{- $gm := index .Values "gitea-mirror" -}}
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: {{ .Values.persistence.claimName }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
{{- with .Values.persistence.annotations }}
annotations:
{{ . | toYaml | indent 4}}
{{- end }}
spec:
accessModes:
{{- toYaml .Values.persistence.accessModes | nindent 4 }}
{{- with .Values.persistence.storageClass }}
storageClassName: {{ . }}
{{- end }}
volumeMode: Filesystem
{{- with .Values.persistence.volumeName }}
volumeName: {{ . }}
{{- end }}
resources:
requests:
storage: {{ .Values.persistence.size }}
{{- end }}

View File

@@ -0,0 +1,14 @@
{{- $gm := index .Values "gitea-mirror" -}}
{{- if (empty $gm.existingSecret) }}
apiVersion: v1
kind: Secret
metadata:
name: {{ include "gitea-mirror.fullname" . }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
type: Opaque
stringData:
GITHUB_TOKEN: {{ $gm.github.token | quote }}
GITEA_TOKEN: {{ $gm.gitea.token | quote }}
ENCRYPTION_SECRET: {{ $gm.core.encryptionSecret | quote }}
{{- end }}

View File

@@ -0,0 +1,34 @@
apiVersion: v1
kind: Service
metadata:
name: {{ include "gitea-mirror.fullname" . }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
{{- if .Values.service.labels }}
{{- toYaml .Values.service.labels | nindent 4 }}
{{- end }}
annotations:
{{- toYaml .Values.service.annotations | nindent 4 }}
spec:
type: {{ .Values.service.type }}
{{- if eq .Values.service.type "LoadBalancer" }}
{{- if .Values.service.loadBalancerClass }}
loadBalancerClass: {{ .Values.service.loadBalancerClass }}
{{- end }}
{{- if and .Values.service.loadBalancerIP }}
loadBalancerIP: {{ .Values.service.loadBalancerIP }}
{{- end }}
{{- end }}
{{- if .Values.service.externalTrafficPolicy }}
externalTrafficPolicy: {{ .Values.service.externalTrafficPolicy }}
{{- end }}
{{- if and .Values.service.clusterIP (eq .Values.service.type "ClusterIP") }}
clusterIP: {{ .Values.service.clusterIP }}
{{- end }}
ports:
- name: http
port: {{ .Values.service.port }}
protocol: TCP
targetPort: http
selector:
{{- include "gitea-mirror.selectorLabels" . | nindent 4 }}

View File

@@ -0,0 +1,17 @@
{{- if .Values.serviceAccount.create }}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "gitea-mirror.serviceAccountName" . }}
labels:
{{- include "gitea-mirror.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.labels }}
{{- . | toYaml | nindent 4 }}
{{- end }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- . | toYaml | nindent 4 }}
{{- end }}
automountServiceAccountToken: {{ .Values.serviceAccount.automountServiceAccountToken }}
{{- end }}

View File

@@ -0,0 +1,151 @@
image:
registry: ghcr.io
repository: raylabshq/gitea-mirror
# Leave blank to use the Appversion tag
tag: ""
pullPolicy: IfNotPresent
imagePullSecrets: []
podSecurityContext:
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
fsGroupChangePolicy: OnRootMismatch
ingress:
enabled: false
className: ""
annotations: {}
hosts:
- host: mirror.example.com
paths:
- path: /
pathType: Prefix
tls: []
# - secretName: chart-example-tls
# hosts:
# - mirror.example.com
route:
enabled: false
forceHTTPS: true
domain: ["mirror.example.com"]
gateway: ""
gatewayNamespace: ""
http:
gatewaySection: ""
filters: []
https:
gatewaySection: ""
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
add:
- name: Strict-Transport-Security
value: "max-age=31536000; includeSubDomains; preload"
service:
type: ClusterIP
port: 8080
clusterIP: None
annotations: {}
externalTrafficPolicy:
labels: {}
loadBalancerIP:
loadBalancerClass:
deployment:
port: 8080
strategy:
type: Recreate
env: []
terminationGracePeriodSeconds: 60
labels: {}
annotations: {}
resources: {}
livenessProbe:
enabled: true
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
readinessProbe:
enabled: true
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
startupProbe:
enabled: true
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 6
successThreshold: 1
persistence:
enabled: true
create: true
claimName: gitea-mirror-storage
storageClass: ""
accessModes:
- ReadWriteOnce
size: 1Gi
affinity: {}
nodeSelector: {}
tolerations: []
topologySpreadConstraints: []
extraVolumes: []
extraVolumeMounts: []
serviceAccount:
create: false
name: ""
annotations: {}
labels: {}
automountServiceAccountToken: false
gitea-mirror:
existingSecret: ""
nodeEnv: production
core:
databaseUrl: file:data/gitea-mirror.db
encryptionSecret: ""
betterAuthSecret: ""
betterAuthUrl: "http://localhost:4321"
betterAuthTrustedOrigins: "http://localhost:4321"
github:
username: ""
token: ""
type: personal
privateRepositories: true
mirrorStarred: false
skipForks: false
starredCodeOnly: false
gitea:
url: ""
token: ""
username: ""
organization: "github-mirrors"
visibility: "public"
mirror:
releases: true
wiki: true
metadata: true
issues: true
pullRequests: true
automation:
schedule_enabled: true
schedule_interval: 3600
cleanup:
enabled: true
retentionDays: 30

View File

@@ -1,7 +1,7 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "3.4.0",
"version": "3.8.7",
"engines": {
"bun": ">=1.2.9"
},
@@ -42,11 +42,12 @@
"devalue": "^5.3.2"
},
"dependencies": {
"@astrojs/check": "^0.9.4",
"@astrojs/mdx": "4.3.4",
"@astrojs/node": "9.4.3",
"@astrojs/react": "^4.3.0",
"@better-auth/sso": "^1.3.7",
"@astrojs/check": "^0.9.5",
"@astrojs/mdx": "4.3.7",
"@astrojs/node": "9.5.0",
"@astrojs/react": "^4.4.0",
"@better-auth/sso": "1.4.0-beta.12",
"@octokit/plugin-throttling": "^11.0.2",
"@octokit/rest": "^22.0.0",
"@radix-ui/react-accordion": "^1.2.12",
"@radix-ui/react-avatar": "^1.1.10",
@@ -57,6 +58,7 @@
"@radix-ui/react-hover-card": "^1.1.15",
"@radix-ui/react-label": "^2.1.7",
"@radix-ui/react-popover": "^1.1.15",
"@radix-ui/react-progress": "^1.1.7",
"@radix-ui/react-radio-group": "^1.3.8",
"@radix-ui/react-scroll-area": "^1.2.10",
"@radix-ui/react-select": "^2.2.6",
@@ -65,48 +67,49 @@
"@radix-ui/react-switch": "^1.2.6",
"@radix-ui/react-tabs": "^1.1.13",
"@radix-ui/react-tooltip": "^1.2.8",
"@tailwindcss/vite": "^4.1.12",
"@tailwindcss/vite": "^4.1.15",
"@tanstack/react-virtual": "^3.13.12",
"@types/canvas-confetti": "^1.9.0",
"@types/react": "^19.1.12",
"@types/react-dom": "^19.1.9",
"astro": "^5.13.4",
"@types/react": "^19.2.2",
"@types/react-dom": "^19.2.2",
"astro": "^5.14.8",
"bcryptjs": "^3.0.2",
"better-auth": "^1.3.7",
"buffer": "^6.0.3",
"better-auth": "1.4.0-beta.12",
"canvas-confetti": "^1.9.3",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"cmdk": "^1.1.1",
"dotenv": "^17.2.1",
"drizzle-orm": "^0.44.5",
"dotenv": "^17.2.3",
"drizzle-orm": "^0.44.6",
"fuse.js": "^7.1.0",
"jsonwebtoken": "^9.0.2",
"lucide-react": "^0.542.0",
"lucide-react": "^0.546.0",
"next-themes": "^0.4.6",
"react": "^19.1.1",
"react-dom": "^19.1.1",
"react": "^19.2.0",
"react-dom": "^19.2.0",
"react-icons": "^5.5.0",
"sonner": "^2.0.7",
"tailwind-merge": "^3.3.1",
"tailwindcss": "^4.1.12",
"tw-animate-css": "^1.3.7",
"typescript": "^5.9.2",
"uuid": "^11.1.0",
"tailwindcss": "^4.1.15",
"tw-animate-css": "^1.4.0",
"typescript": "^5.9.3",
"uuid": "^13.0.0",
"vaul": "^1.1.2",
"zod": "^4.1.5"
"zod": "^4.1.12"
},
"devDependencies": {
"@testing-library/jest-dom": "^6.8.0",
"@testing-library/jest-dom": "^6.9.1",
"@testing-library/react": "^16.3.0",
"@types/bcryptjs": "^3.0.0",
"@types/bun": "^1.2.21",
"@types/bun": "^1.3.0",
"@types/jsonwebtoken": "^9.0.10",
"@types/uuid": "^10.0.0",
"@vitejs/plugin-react": "^5.0.2",
"drizzle-kit": "^0.31.4",
"@vitejs/plugin-react": "^5.0.4",
"drizzle-kit": "^0.31.5",
"jsdom": "^26.1.0",
"tsx": "^4.20.5",
"tsx": "^4.20.6",
"vitest": "^3.2.4"
},
"packageManager": "bun@1.2.21"
"packageManager": "bun@1.2.23"
}

View File

@@ -67,21 +67,21 @@ export function AdvancedOptionsForm({
<div className="flex items-center">
<Checkbox
id="skip-starred-issues"
checked={config.skipStarredIssues}
id="starred-code-only"
checked={config.starredCodeOnly}
onCheckedChange={(checked) =>
handleChange("skipStarredIssues", Boolean(checked))
handleChange("starredCodeOnly", Boolean(checked))
}
/>
<label
htmlFor="skip-starred-issues"
htmlFor="starred-code-only"
className="ml-2 text-sm select-none"
>
Don't fetch issues for starred repos
Code-only mode for starred repos
</label>
</div>
<p className="text-xs text-muted-foreground ml-6">
Skip mirroring issues and pull requests for starred repositories
Mirror only source code for starred repositories, skipping all metadata (issues, PRs, labels, milestones, wiki, releases)
</p>
</div>
</CardContent>

View File

@@ -9,6 +9,7 @@ import {
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import { Switch } from "@/components/ui/switch";
import {
Clock,
Database,
@@ -16,7 +17,8 @@ import {
Calendar,
Activity,
Zap,
Info
Info,
Archive,
} from "lucide-react";
import {
Tooltip,
@@ -120,14 +122,14 @@ export function AutomationSettings({
</CardTitle>
</CardHeader>
<CardContent className="space-y-6">
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
{/* Automatic Mirroring Section */}
<div className="space-y-4 p-4 border border-border rounded-lg bg-card/50">
<div className="flex items-center justify-between">
<h3 className="text-sm font-medium flex items-center gap-2">
<RefreshCw className="h-4 w-4 text-primary" />
Automatic Mirroring
<CardContent className="space-y-6">
<div className="grid grid-cols-1 md:grid-cols-2 gap-4">
{/* Automatic Syncing Section */}
<div className="space-y-4 p-4 border border-border rounded-lg bg-card/50">
<div className="flex items-center justify-between">
<h3 className="text-sm font-medium flex items-center gap-2">
<RefreshCw className="h-4 w-4 text-primary" />
Automatic Syncing
</h3>
{isAutoSavingSchedule && (
<Activity className="h-4 w-4 animate-spin text-muted-foreground" />
@@ -139,6 +141,7 @@ export function AutomationSettings({
<Checkbox
id="enable-auto-mirror"
checked={scheduleConfig.enabled}
className="mt-1.25"
onCheckedChange={(checked) =>
onScheduleChange({ ...scheduleConfig, enabled: !!checked })
}
@@ -218,17 +221,17 @@ export function AutomationSettings({
Enable automatic syncing to schedule periodic repository updates
</div>
)}
</div>
</div>
</div>
</div>
</div>
{/* Database Cleanup Section */}
<div className="space-y-4 p-4 border border-border rounded-lg bg-card/50">
<div className="flex items-center justify-between">
<h3 className="text-sm font-medium flex items-center gap-2">
<Database className="h-4 w-4 text-primary" />
Database Maintenance
</h3>
{/* Database Cleanup Section */}
<div className="space-y-4 p-4 border border-border rounded-lg bg-card/50">
<div className="flex items-center justify-between">
<h3 className="text-sm font-medium flex items-center gap-2">
<Database className="h-4 w-4 text-primary" />
Database Maintenance
</h3>
{isAutoSavingCleanup && (
<Activity className="h-4 w-4 animate-spin text-muted-foreground" />
)}
@@ -239,6 +242,7 @@ export function AutomationSettings({
<Checkbox
id="enable-auto-cleanup"
checked={cleanupConfig.enabled}
className="mt-1.25"
onCheckedChange={(checked) =>
onCleanupChange({ ...cleanupConfig, enabled: !!checked })
}
@@ -257,8 +261,8 @@ export function AutomationSettings({
</div>
{cleanupConfig.enabled && (
<div className="ml-6 space-y-3">
<div>
<div className="ml-6 space-y-5">
<div className="space-y-2">
<Label htmlFor="retention-period" className="text-sm flex items-center gap-2">
Data retention period
<TooltipProvider>
@@ -275,35 +279,36 @@ export function AutomationSettings({
</Tooltip>
</TooltipProvider>
</Label>
<Select
value={cleanupConfig.retentionDays.toString()}
onValueChange={(value) =>
onCleanupChange({
...cleanupConfig,
retentionDays: parseInt(value, 10),
})
}
>
<SelectTrigger id="retention-period" className="mt-1.5">
<SelectValue />
</SelectTrigger>
<SelectContent>
{retentionPeriods.map((option) => (
<SelectItem
key={option.value}
value={option.value.toString()}
>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
{cleanupConfig.enabled && (
<p className="text-xs text-muted-foreground mt-1">
<div className="flex items-center gap-3 mt-1.5">
<Select
value={cleanupConfig.retentionDays.toString()}
onValueChange={(value) =>
onCleanupChange({
...cleanupConfig,
retentionDays: parseInt(value, 10),
})
}
>
<SelectTrigger id="retention-period" className="w-40">
<SelectValue />
</SelectTrigger>
<SelectContent>
{retentionPeriods.map((option) => (
<SelectItem
key={option.value}
value={option.value.toString()}
>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Cleanup runs {getCleanupFrequencyText(cleanupConfig.retentionDays)}
</p>
)}
</div>
</div>
</div>
)}
@@ -334,13 +339,108 @@ export function AutomationSettings({
) : (
<div className="text-xs text-muted-foreground">
Enable automatic cleanup to optimize database storage
</div>
)}
</div>
</div>
)}
</div>
</div>
</CardContent>
</div>
{/* Repository Cleanup Section */}
<div className="space-y-4 p-4 border border-border rounded-lg bg-card/50 md:col-span-2">
<div className="flex items-center justify-between">
<h3 className="text-sm font-medium flex items-center gap-2">
<Archive className="h-4 w-4 text-primary" />
Repository Cleanup (orphaned mirrors)
</h3>
{isAutoSavingCleanup && (
<Activity className="h-4 w-4 animate-spin text-muted-foreground" />
)}
</div>
<div className="space-y-4">
<div className="flex items-start space-x-3">
<Checkbox
id="cleanup-handle-orphans"
checked={Boolean(cleanupConfig.deleteIfNotInGitHub)}
className="mt-1.25"
onCheckedChange={(checked) =>
onCleanupChange({
...cleanupConfig,
deleteIfNotInGitHub: Boolean(checked),
})
}
/>
<div className="space-y-0.5 flex-1">
<Label
htmlFor="cleanup-handle-orphans"
className="text-sm font-normal cursor-pointer"
>
Handle orphaned repositories automatically
</Label>
<p className="text-xs text-muted-foreground">
Keep your Gitea backups when GitHub repos disappear. Archive is the safest optionit preserves data and disables automatic syncs.
</p>
</div>
</div>
{cleanupConfig.deleteIfNotInGitHub && (
<div className="space-y-3 ml-6">
<div className="space-y-1">
<Label htmlFor="cleanup-orphaned-action" className="text-sm font-medium">
Action for orphaned repositories
</Label>
<Select
value={cleanupConfig.orphanedRepoAction ?? "archive"}
onValueChange={(value) =>
onCleanupChange({
...cleanupConfig,
orphanedRepoAction: value as DatabaseCleanupConfig["orphanedRepoAction"],
})
}
>
<SelectTrigger id="cleanup-orphaned-action">
<SelectValue />
</SelectTrigger>
<SelectContent>
<SelectItem value="archive">Archive (preserve data)</SelectItem>
<SelectItem value="skip">Skip (leave as-is)</SelectItem>
<SelectItem value="delete">Delete from Gitea</SelectItem>
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground">
Archive renames mirror backups with an <code>archived-</code> prefix and disables automatic syncsuse Manual Sync when you want to refresh.
</p>
</div>
<div className="flex items-center justify-between">
<div className="space-y-0.5">
<Label
htmlFor="cleanup-dry-run"
className="text-sm font-normal cursor-pointer"
>
Dry run (log only)
</Label>
<p className="text-xs text-muted-foreground max-w-xl">
When enabled, cleanup logs the planned action without modifying repositories.
</p>
</div>
<Switch
id="cleanup-dry-run"
checked={Boolean(cleanupConfig.dryRun)}
onCheckedChange={(checked) =>
onCleanupChange({
...cleanupConfig,
dryRun: Boolean(checked),
})
}
/>
</div>
</div>
)}
</div>
</div>
</div>
</CardContent>
</Card>
);
}
}

View File

@@ -50,12 +50,17 @@ export function ConfigTabs() {
preserveOrgStructure: false,
},
scheduleConfig: {
enabled: true, // Default to enabled
interval: 86400, // Default to daily (24 hours)
enabled: false, // Don't set defaults here - will be loaded from API
interval: 0, // Will be replaced with actual value from API
},
cleanupConfig: {
enabled: true, // Default to enabled
retentionDays: 604800, // 7 days in seconds - Default retention period
enabled: false, // Don't set defaults here - will be loaded from API
retentionDays: 0, // Will be replaced with actual value from API
deleteIfNotInGitHub: true,
orphanedRepoAction: "archive",
dryRun: false,
deleteFromGitea: false,
protectedRepos: [],
},
mirrorOptions: {
mirrorReleases: false,
@@ -71,7 +76,7 @@ export function ConfigTabs() {
},
advancedOptions: {
skipForks: false,
skipStarredIssues: false,
starredCodeOnly: false,
},
});
const { user } = useAuth();

View File

@@ -1,201 +0,0 @@
import { Card, CardContent } from "@/components/ui/card";
import { Checkbox } from "../ui/checkbox";
import type { DatabaseCleanupConfig } from "@/types/config";
import { formatDate } from "@/lib/utils";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "../ui/select";
import { RefreshCw, Database } from "lucide-react";
interface DatabaseCleanupConfigFormProps {
config: DatabaseCleanupConfig;
setConfig: React.Dispatch<React.SetStateAction<DatabaseCleanupConfig>>;
onAutoSave?: (config: DatabaseCleanupConfig) => void;
isAutoSaving?: boolean;
}
// Helper to calculate cleanup interval in hours (should match backend logic)
function calculateCleanupInterval(retentionSeconds: number): number {
const retentionDays = retentionSeconds / (24 * 60 * 60);
if (retentionDays <= 1) {
return 6;
} else if (retentionDays <= 3) {
return 12;
} else if (retentionDays <= 7) {
return 24;
} else if (retentionDays <= 30) {
return 48;
} else {
return 168;
}
}
export function DatabaseCleanupConfigForm({
config,
setConfig,
onAutoSave,
isAutoSaving = false,
}: DatabaseCleanupConfigFormProps) {
// Optimistically update nextRun when enabled or retention changes
const handleChange = (
e: React.ChangeEvent<HTMLInputElement | HTMLSelectElement>
) => {
const { name, value, type } = e.target;
let newConfig = {
...config,
[name]: type === "checkbox" ? (e.target as HTMLInputElement).checked : value,
};
// If enabling or changing retention, recalculate nextRun
if (
(name === "enabled" && (e.target as HTMLInputElement).checked) ||
(name === "retentionDays" && config.enabled)
) {
const now = new Date();
const retentionSeconds =
name === "retentionDays"
? Number(value)
: Number(newConfig.retentionDays);
const intervalHours = calculateCleanupInterval(retentionSeconds);
const nextRun = new Date(now.getTime() + intervalHours * 60 * 60 * 1000);
newConfig = {
...newConfig,
nextRun,
};
}
// If disabling, clear nextRun
if (name === "enabled" && !(e.target as HTMLInputElement).checked) {
newConfig = {
...newConfig,
nextRun: undefined,
};
}
setConfig(newConfig);
if (onAutoSave) {
onAutoSave(newConfig);
}
};
// Predefined retention periods (in seconds, like schedule intervals)
const retentionOptions: { value: number; label: string }[] = [
{ value: 86400, label: "1 day" }, // 24 * 60 * 60
{ value: 259200, label: "3 days" }, // 3 * 24 * 60 * 60
{ value: 604800, label: "7 days" }, // 7 * 24 * 60 * 60
{ value: 1209600, label: "14 days" }, // 14 * 24 * 60 * 60
{ value: 2592000, label: "30 days" }, // 30 * 24 * 60 * 60
{ value: 5184000, label: "60 days" }, // 60 * 24 * 60 * 60
{ value: 7776000, label: "90 days" }, // 90 * 24 * 60 * 60
];
return (
<Card className="self-start">
<CardContent className="pt-6 relative">
{isAutoSaving && (
<div className="absolute top-4 right-4 flex items-center text-sm text-muted-foreground">
<RefreshCw className="h-3 w-3 animate-spin mr-1" />
<span className="text-xs">Auto-saving...</span>
</div>
)}
<div className="flex flex-col gap-y-4">
<div className="flex items-center">
<Checkbox
id="cleanup-enabled"
name="enabled"
checked={config.enabled}
onCheckedChange={(checked) =>
handleChange({
target: {
name: "enabled",
type: "checkbox",
checked: Boolean(checked),
value: "",
},
} as React.ChangeEvent<HTMLInputElement>)
}
/>
<label
htmlFor="cleanup-enabled"
className="select-none ml-2 block text-sm font-medium"
>
<div className="flex items-center gap-2">
<Database className="h-4 w-4" />
Enable Automatic Database Cleanup
</div>
</label>
</div>
{config.enabled && (
<div>
<label className="block text-sm font-medium mb-2">
Data Retention Period
</label>
<Select
name="retentionDays"
value={String(config.retentionDays)}
onValueChange={(value) =>
handleChange({
target: { name: "retentionDays", value },
} as React.ChangeEvent<HTMLInputElement>)
}
>
<SelectTrigger className="w-full border border-input dark:bg-background dark:hover:bg-background">
<SelectValue placeholder="Select retention period" />
</SelectTrigger>
<SelectContent className="bg-background text-foreground border border-input shadow-sm">
{retentionOptions.map((option) => (
<SelectItem
key={option.value}
value={option.value.toString()}
className="cursor-pointer text-sm px-3 py-2 hover:bg-accent focus:bg-accent focus:text-accent-foreground"
>
{option.label}
</SelectItem>
))}
</SelectContent>
</Select>
<p className="text-xs text-muted-foreground mt-1">
Activities and events older than this period will be automatically deleted.
</p>
<div className="mt-2 p-2 bg-muted/50 rounded-md">
<p className="text-xs text-muted-foreground">
<strong>Cleanup Frequency:</strong> The cleanup process runs automatically at optimal intervals:
shorter retention periods trigger more frequent cleanups, longer periods trigger less frequent cleanups.
</p>
</div>
</div>
)}
<div className="flex gap-x-4">
<div className="flex-1">
<label className="block text-sm font-medium mb-1">Last Cleanup</label>
<div className="text-sm">
{config.lastRun ? formatDate(config.lastRun) : "Never"}
</div>
</div>
{config.enabled && (
<div className="flex-1">
<label className="block text-sm font-medium mb-1">Next Cleanup</label>
<div className="text-sm">
{config.nextRun
? formatDate(config.nextRun)
: config.enabled
? "Calculating..."
: "Never"}
</div>
</div>
)}
</div>
</div>
</CardContent>
</Card>
);
}

View File

@@ -15,11 +15,11 @@ import {
PopoverContent,
PopoverTrigger,
} from "@/components/ui/popover";
import {
Info,
GitBranch,
Star,
Lock,
import {
Info,
GitBranch,
Star,
Lock,
Archive,
GitPullRequest,
Tag,
@@ -30,9 +30,17 @@ import {
GitFork,
ChevronDown,
Funnel,
HardDrive
HardDrive,
FileCode2
} from "lucide-react";
import type { GitHubConfig, MirrorOptions, AdvancedOptions } from "@/types/config";
import type { GitHubConfig, MirrorOptions, AdvancedOptions, DuplicateNameStrategy } from "@/types/config";
import {
Select,
SelectContent,
SelectItem,
SelectTrigger,
SelectValue,
} from "@/components/ui/select";
import { cn } from "@/lib/utils";
interface GitHubMirrorSettingsProps {
@@ -53,7 +61,7 @@ export function GitHubMirrorSettings({
onAdvancedOptionsChange,
}: GitHubMirrorSettingsProps) {
const handleGitHubChange = (field: keyof GitHubConfig, value: boolean) => {
const handleGitHubChange = (field: keyof GitHubConfig, value: boolean | string) => {
onGitHubConfigChange({ ...githubConfig, [field]: value });
};
@@ -81,10 +89,10 @@ export function GitHubMirrorSettings({
// Calculate what content is included for starred repos
const starredRepoContent = {
code: true, // Always included
releases: !advancedOptions.skipStarredIssues && mirrorOptions.mirrorReleases,
issues: !advancedOptions.skipStarredIssues && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.issues,
pullRequests: !advancedOptions.skipStarredIssues && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.pullRequests,
wiki: !advancedOptions.skipStarredIssues && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.wiki,
releases: !advancedOptions.starredCodeOnly && mirrorOptions.mirrorReleases,
issues: !advancedOptions.starredCodeOnly && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.issues,
pullRequests: !advancedOptions.starredCodeOnly && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.pullRequests,
wiki: !advancedOptions.starredCodeOnly && mirrorOptions.mirrorMetadata && mirrorOptions.metadataComponents.wiki,
};
const starredContentCount = Object.entries(starredRepoContent).filter(([key, value]) => key !== 'code' && value).length;
@@ -160,7 +168,7 @@ export function GitHubMirrorSettings({
className="h-8 text-xs font-normal min-w-[140px] md:min-w-[140px] justify-between"
>
<span>
{advancedOptions.skipStarredIssues ? (
{advancedOptions.starredCodeOnly ? (
"Code only"
) : starredContentCount === 0 ? (
"Code only"
@@ -198,8 +206,8 @@ export function GitHubMirrorSettings({
<div className="flex items-center space-x-3 py-1 px-1 rounded hover:bg-accent">
<Checkbox
id="starred-lightweight"
checked={advancedOptions.skipStarredIssues}
onCheckedChange={(checked) => handleAdvancedChange('skipStarredIssues', !!checked)}
checked={advancedOptions.starredCodeOnly}
onCheckedChange={(checked) => handleAdvancedChange('starredCodeOnly', !!checked)}
/>
<Label
htmlFor="starred-lightweight"
@@ -214,7 +222,7 @@ export function GitHubMirrorSettings({
</Label>
</div>
{!advancedOptions.skipStarredIssues && (
{!advancedOptions.starredCodeOnly && (
<>
<Separator className="my-2" />
<div className="space-y-2">
@@ -278,6 +286,40 @@ export function GitHubMirrorSettings({
</Popover>
</div>
</div>
{/* Duplicate name handling for starred repos */}
{githubConfig.mirrorStarred && (
<div className="mt-4 space-y-2">
<Label className="text-xs font-medium text-muted-foreground">
Duplicate name handling
</Label>
<div className="flex items-center gap-3">
<FileCode2 className="h-4 w-4 text-muted-foreground" />
<div className="flex-1">
<p className="text-sm">Name collision strategy</p>
<p className="text-xs text-muted-foreground">
How to handle repos with the same name from different owners
</p>
</div>
<Select
value={githubConfig.starredDuplicateStrategy || "suffix"}
onValueChange={(value) => handleGitHubChange('starredDuplicateStrategy', value as DuplicateNameStrategy)}
>
<SelectTrigger className="w-[180px] h-8 text-xs">
<SelectValue placeholder="Select strategy" />
</SelectTrigger>
<SelectContent align="end">
<SelectItem value="suffix" className="text-xs">
<span className="font-mono">repo-owner</span>
</SelectItem>
<SelectItem value="prefix" className="text-xs">
<span className="font-mono">owner-repo</span>
</SelectItem>
</SelectContent>
</Select>
</div>
</div>
)}
</div>
</div>
@@ -596,4 +638,4 @@ export function GitHubMirrorSettings({
</div>
</div>
);
}
}

View File

@@ -372,8 +372,8 @@ export function SSOSettings() {
Add Provider
</Button>
</DialogTrigger>
<DialogContent className="max-w-2xl">
<DialogHeader>
<DialogContent className="max-w-2xl max-h-[90vh] md:max-h-[85vh] lg:max-h-[90vh] overflow-hidden flex flex-col">
<DialogHeader className="flex-shrink-0">
<DialogTitle>{editingProvider ? 'Edit SSO Provider' : 'Add SSO Provider'}</DialogTitle>
<DialogDescription>
{editingProvider
@@ -381,14 +381,15 @@ export function SSOSettings() {
: 'Configure an external identity provider for user authentication'}
</DialogDescription>
</DialogHeader>
<Tabs value={providerType} onValueChange={(value) => setProviderType(value as 'oidc' | 'saml')}>
<TabsList className="grid w-full grid-cols-2">
<TabsTrigger value="oidc">OIDC / OAuth2</TabsTrigger>
<TabsTrigger value="saml">SAML 2.0</TabsTrigger>
</TabsList>
{/* Common Fields */}
<div className="space-y-4 mt-4">
<div className="flex-1 overflow-y-auto px-1 -mx-1">
<Tabs value={providerType} onValueChange={(value) => setProviderType(value as 'oidc' | 'saml')}>
<TabsList className="grid w-full grid-cols-2 sticky top-0 z-10 bg-background">
<TabsTrigger value="oidc">OIDC / OAuth2</TabsTrigger>
<TabsTrigger value="saml">SAML 2.0</TabsTrigger>
</TabsList>
{/* Common Fields */}
<div className="space-y-4 mt-4">
<div className="grid grid-cols-2 gap-4">
<div className="space-y-2">
<Label htmlFor="providerId">Provider ID</Label>
@@ -569,7 +570,8 @@ export function SSOSettings() {
</Alert>
</TabsContent>
</Tabs>
<DialogFooter>
</div>
<DialogFooter className="flex-shrink-0 pt-4 border-t">
<Button
variant="outline"
onClick={() => {

View File

@@ -1,47 +0,0 @@
import React from 'react';
import { ScheduleConfigForm } from './ScheduleConfigForm';
import { DatabaseCleanupConfigForm } from './DatabaseCleanupConfigForm';
import { Separator } from '../ui/separator';
import type { ScheduleConfig, DatabaseCleanupConfig } from '@/types/config';
interface ScheduleAndCleanupFormProps {
scheduleConfig: ScheduleConfig;
cleanupConfig: DatabaseCleanupConfig;
setScheduleConfig: (update: ScheduleConfig | ((prev: ScheduleConfig) => ScheduleConfig)) => void;
setCleanupConfig: (update: DatabaseCleanupConfig | ((prev: DatabaseCleanupConfig) => DatabaseCleanupConfig)) => void;
onAutoSaveSchedule?: (config: ScheduleConfig) => Promise<void>;
onAutoSaveCleanup?: (config: DatabaseCleanupConfig) => Promise<void>;
isAutoSavingSchedule?: boolean;
isAutoSavingCleanup?: boolean;
}
export function ScheduleAndCleanupForm({
scheduleConfig,
cleanupConfig,
setScheduleConfig,
setCleanupConfig,
onAutoSaveSchedule,
onAutoSaveCleanup,
isAutoSavingSchedule,
isAutoSavingCleanup,
}: ScheduleAndCleanupFormProps) {
return (
<div className="space-y-6">
<ScheduleConfigForm
config={scheduleConfig}
setConfig={setScheduleConfig}
onAutoSave={onAutoSaveSchedule}
isAutoSaving={isAutoSavingSchedule}
/>
<Separator />
<DatabaseCleanupConfigForm
config={cleanupConfig}
setConfig={setCleanupConfig}
onAutoSave={onAutoSaveCleanup}
isAutoSaving={isAutoSavingCleanup}
/>
</div>
);
}

View File

@@ -83,7 +83,7 @@ export function ScheduleConfigForm({
htmlFor="enabled"
className="select-none ml-2 block text-sm font-medium"
>
Enable Automatic Mirroring
Enable Automatic Syncing
</label>
</div>
@@ -93,7 +93,7 @@ export function ScheduleConfigForm({
htmlFor="interval"
className="block text-sm font-medium mb-1.5"
>
Mirroring Interval
Sync Interval
</label>
<Select
@@ -122,7 +122,7 @@ export function ScheduleConfigForm({
</Select>
<p className="text-xs text-muted-foreground mt-1">
How often the mirroring process should run.
How often the sync process should run.
</p>
<div className="mt-2 p-2 bg-muted/50 rounded-md">
<p className="text-xs text-muted-foreground">

View File

@@ -9,6 +9,7 @@ import { apiRequest, showErrorToast } from "@/lib/utils";
import type { DashboardApiResponse } from "@/types/dashboard";
import { useSSE } from "@/hooks/useSEE";
import { toast } from "sonner";
import { useEffect as useEffectForToasts } from "react";
import { Skeleton } from "@/components/ui/skeleton";
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
@@ -16,6 +17,46 @@ import { usePageVisibility } from "@/hooks/usePageVisibility";
import { useConfigStatus } from "@/hooks/useConfigStatus";
import { useNavigation } from "@/components/layout/MainLayout";
// Helper function to format last sync time
function formatLastSyncTime(date: Date | null): string {
if (!date) return "Never";
const now = new Date();
const syncDate = new Date(date);
const diffMs = now.getTime() - syncDate.getTime();
const diffMins = Math.floor(diffMs / 60000);
const diffHours = Math.floor(diffMs / 3600000);
const diffDays = Math.floor(diffMs / 86400000);
// Show relative time for recent syncs
if (diffMins < 1) return "Just now";
if (diffMins < 60) return `${diffMins} min ago`;
if (diffHours < 24) return `${diffHours} hr${diffHours === 1 ? '' : 's'} ago`;
if (diffDays < 7) return `${diffDays} day${diffDays === 1 ? '' : 's'} ago`;
// For older syncs, show week count
const diffWeeks = Math.floor(diffDays / 7);
if (diffWeeks < 4) return `${diffWeeks} week${diffWeeks === 1 ? '' : 's'} ago`;
// For even older, show month count
const diffMonths = Math.floor(diffDays / 30);
return `${diffMonths} month${diffMonths === 1 ? '' : 's'} ago`;
}
// Helper function to format full timestamp
function formatFullTimestamp(date: Date | null): string {
if (!date) return "";
return new Date(date).toLocaleString("en-US", {
month: "2-digit",
day: "2-digit",
year: "2-digit",
hour: "2-digit",
minute: "2-digit",
hour12: true
}).replace(',', '');
}
export function Dashboard() {
const { user } = useAuth();
const { registerRefreshCallback } = useLiveRefresh();
@@ -65,6 +106,51 @@ export function Dashboard() {
onMessage: handleNewMessage,
});
// Setup rate limit event listener for toast notifications
useEffectForToasts(() => {
if (!user?.id) return;
const eventSource = new EventSource(`/api/events?userId=${user.id}`);
eventSource.addEventListener("rate-limit", (event) => {
try {
const data = JSON.parse(event.data);
switch (data.type) {
case "warning":
// 80% threshold warning
toast.warning("GitHub API Rate Limit Warning", {
description: data.message,
duration: 8000,
});
break;
case "exceeded":
// 100% rate limit exceeded
toast.error("GitHub API Rate Limit Exceeded", {
description: data.message,
duration: 10000,
});
break;
case "resumed":
// Rate limit reset notification
toast.success("Rate Limit Reset", {
description: "API operations have resumed.",
duration: 5000,
});
break;
}
} catch (error) {
console.error("Error parsing rate limit event:", error);
}
});
return () => {
eventSource.close();
};
}, [user?.id]);
// Extract fetchDashboardData as a stable callback
const fetchDashboardData = useCallback(async (showToast = false) => {
try {
@@ -236,19 +322,9 @@ export function Dashboard() {
/>
<StatusCard
title="Last Sync"
value={
lastSync
? new Date(lastSync).toLocaleString("en-US", {
month: "2-digit",
day: "2-digit",
year: "2-digit",
hour: "2-digit",
minute: "2-digit",
})
: "N/A"
}
value={formatLastSyncTime(lastSync)}
icon={<Clock className="h-4 w-4" />}
description="Last successful sync"
description={formatFullTimestamp(lastSync)}
/>
</div>

View File

@@ -7,7 +7,7 @@ import { toast } from "sonner";
import { Skeleton } from "@/components/ui/skeleton";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
import { useConfigStatus } from "@/hooks/useConfigStatus";
import { Menu, LogOut } from "lucide-react";
import { Menu, LogOut, PanelRightOpen, PanelRightClose } from "lucide-react";
import {
DropdownMenu,
DropdownMenuContent,
@@ -19,9 +19,12 @@ interface HeaderProps {
currentPage?: "dashboard" | "repositories" | "organizations" | "configuration" | "activity-log";
onNavigate?: (page: string) => void;
onMenuClick: () => void;
onToggleCollapse?: () => void;
isSidebarCollapsed?: boolean;
isSidebarOpen?: boolean;
}
export function Header({ currentPage, onNavigate, onMenuClick }: HeaderProps) {
export function Header({ currentPage, onNavigate, onMenuClick, onToggleCollapse, isSidebarCollapsed, isSidebarOpen }: HeaderProps) {
const { user, logout, isLoading } = useAuth();
const { isLiveEnabled, toggleLive } = useLiveRefresh();
const { isFullyConfigured, isLoading: configLoading } = useConfigStatus();
@@ -63,18 +66,38 @@ export function Header({ currentPage, onNavigate, onMenuClick }: HeaderProps) {
return (
<header className="border-b bg-background">
<div className="flex h-[4.5rem] items-center justify-between px-4 sm:px-6">
<div className="flex items-center gap-2">
{/* Hamburger Menu Button - Mobile Only */}
<div className="flex items-center lg:gap-12 md:gap-6 gap-4">
{/* Sidebar Toggle - Mobile uses slide-in, Medium uses collapse */}
<Button
variant="outline"
size="lg"
className="lg:hidden"
size="icon"
className="md:hidden h-10 w-10"
onClick={onMenuClick}
>
<Menu className="h-5 w-5" />
{isSidebarOpen ? (
<PanelRightOpen className="h-5 w-5" />
) : (
<PanelRightClose className="h-5 w-5" />
)}
<span className="sr-only">Toggle menu</span>
</Button>
{/* Sidebar Collapse Toggle - Only on medium screens (768px - 1280px) */}
<Button
variant="ghost"
size="icon"
className="hidden md:flex xl:hidden h-10 w-10"
onClick={onToggleCollapse}
title={isSidebarCollapsed ? "Expand sidebar" : "Collapse sidebar"}
>
{isSidebarCollapsed ? (
<PanelRightClose className="h-5 w-5" />
) : (
<PanelRightOpen className="h-5 w-5" />
)}
<span className="sr-only">Toggle sidebar</span>
</Button>
<button
onClick={() => {
if (currentPage !== 'dashboard') {

View File

@@ -45,6 +45,13 @@ function AppWithProviders({ page: initialPage }: AppProps) {
const [currentPage, setCurrentPage] = useState<AppProps['page']>(initialPage);
const [navigationKey, setNavigationKey] = useState(0);
const [sidebarOpen, setSidebarOpen] = useState(false);
const [sidebarCollapsed, setSidebarCollapsed] = useState(() => {
// Check if we're on medium screens (768px - 1280px)
if (typeof window !== 'undefined') {
return window.innerWidth >= 768 && window.innerWidth < 1280;
}
return false;
});
useRepoSync({
userId: user?.id,
@@ -83,6 +90,23 @@ function AppWithProviders({ page: initialPage }: AppProps) {
return () => window.removeEventListener('popstate', handlePopState);
}, []);
// Handle window resize to auto-collapse sidebar on medium screens
useEffect(() => {
const handleResize = () => {
const width = window.innerWidth;
// Auto-collapse on medium screens (768px - 1280px)
if (width >= 768 && width < 1280) {
setSidebarCollapsed(true);
} else if (width >= 1280) {
// Expand on large screens
setSidebarCollapsed(false);
}
};
window.addEventListener('resize', handleResize);
return () => window.removeEventListener('resize', handleResize);
}, []);
// Show loading state only during initial auth/config loading
const isInitialLoading = authLoading || (configLoading && !user);
@@ -113,14 +137,21 @@ function AppWithProviders({ page: initialPage }: AppProps) {
currentPage={currentPage}
onNavigate={handleNavigation}
onMenuClick={() => setSidebarOpen(!sidebarOpen)}
onToggleCollapse={() => setSidebarCollapsed(!sidebarCollapsed)}
isSidebarCollapsed={sidebarCollapsed}
isSidebarOpen={sidebarOpen}
/>
<div className="flex flex-1 relative">
<Sidebar
onNavigate={handleNavigation}
isOpen={sidebarOpen}
isCollapsed={sidebarCollapsed}
onClose={() => setSidebarOpen(false)}
onToggleCollapse={() => setSidebarCollapsed(!sidebarCollapsed)}
/>
<section className="flex-1 p-4 sm:p-6 overflow-y-auto h-[calc(100dvh-4.55rem)] w-full lg:w-[calc(100%-16rem)]">
<section className={`flex-1 p-4 sm:p-6 overflow-y-auto h-[calc(100dvh-4.55rem)] w-full transition-all duration-200 ${
sidebarCollapsed ? 'md:w-[calc(100%-5rem)] xl:w-[calc(100%-16rem)]' : 'md:w-[calc(100%-16rem)]'
}`}>
{currentPage === "dashboard" && <Dashboard />}
{currentPage === "repositories" && <Repository />}
{currentPage === "organizations" && <Organization />}

View File

@@ -3,15 +3,23 @@ import { cn } from "@/lib/utils";
import { ExternalLink } from "lucide-react";
import { links } from "@/data/Sidebar";
import { VersionInfo } from "./VersionInfo";
import {
Tooltip,
TooltipContent,
TooltipProvider,
TooltipTrigger,
} from "@/components/ui/tooltip";
interface SidebarProps {
className?: string;
onNavigate?: (page: string) => void;
isOpen: boolean;
isCollapsed?: boolean;
onClose: () => void;
onToggleCollapse?: () => void;
}
export function Sidebar({ className, onNavigate, isOpen, onClose }: SidebarProps) {
export function Sidebar({ className, onNavigate, isOpen, isCollapsed = false, onClose, onToggleCollapse }: SidebarProps) {
const [currentPath, setCurrentPath] = useState<string>("");
useEffect(() => {
@@ -53,7 +61,7 @@ export function Sidebar({ className, onNavigate, isOpen, onClose }: SidebarProps
onNavigate?.(pageName);
// Close sidebar on mobile after navigation
if (window.innerWidth < 1024) {
if (window.innerWidth < 768) {
onClose();
}
};
@@ -63,7 +71,7 @@ export function Sidebar({ className, onNavigate, isOpen, onClose }: SidebarProps
{/* Mobile Backdrop */}
{isOpen && (
<div
className="fixed inset-0 backdrop-blur-sm z-40 lg:hidden"
className="fixed inset-0 backdrop-blur-sm z-40 md:hidden"
onClick={onClose}
/>
)}
@@ -71,54 +79,126 @@ export function Sidebar({ className, onNavigate, isOpen, onClose }: SidebarProps
{/* Sidebar */}
<aside
className={cn(
"fixed lg:static inset-y-0 left-0 z-50 w-64 bg-background border-r flex flex-col h-full lg:h-[calc(100vh-4.5rem)] transition-transform duration-200 ease-in-out lg:translate-x-0",
"fixed md:static inset-y-0 left-0 z-50 bg-background border-r flex flex-col h-full md:h-[calc(100vh-4.5rem)] transition-all duration-200 ease-in-out md:translate-x-0",
isOpen ? "translate-x-0" : "-translate-x-full",
isCollapsed ? "md:w-20 xl:w-64" : "w-64",
className
)}
>
<div className="flex flex-col h-full">
<nav className="flex flex-col gap-y-1 lg:gap-y-1 pl-2 pr-3 pt-4 flex-shrink-0">
<nav className={cn(
"flex flex-col pt-4 flex-shrink-0",
isCollapsed
? "md:gap-y-2 md:items-center md:px-2 xl:gap-y-1 xl:items-stretch xl:pl-2 xl:pr-3 gap-y-1 pl-2 pr-3"
: "gap-y-1 pl-2 pr-3"
)}>
{links.map((link, index) => {
const isActive = currentPath === link.href;
const Icon = link.icon;
return (
const button = (
<button
key={index}
onClick={(e) => handleNavigation(link.href, e)}
className={cn(
"flex items-center gap-3 rounded-md px-3 py-3 lg:py-2 text-sm lg:text-sm font-medium transition-colors w-full text-left",
"flex items-center rounded-md text-sm font-medium transition-colors w-full",
isCollapsed
? "md:h-12 md:w-12 md:justify-center md:p-0 xl:h-auto xl:w-full xl:justify-start xl:px-3 xl:py-2 h-auto px-3 py-3"
: "px-3 py-3 md:py-2",
isActive
? "bg-primary text-primary-foreground"
: "text-muted-foreground hover:bg-accent hover:text-accent-foreground"
)}
>
<Icon className="h-5 w-5 lg:h-4 lg:w-4" />
{link.label}
<Icon className={cn(
"flex-shrink-0",
isCollapsed
? "md:h-5 md:w-5 md:mr-0 xl:h-4 xl:w-4 xl:mr-3 h-5 w-5 mr-3"
: "h-5 w-5 md:h-4 md:w-4 mr-3"
)} />
<span className={cn(
"transition-all duration-200",
isCollapsed ? "md:hidden xl:inline" : "inline"
)}>
{link.label}
</span>
</button>
);
// Wrap in tooltip when collapsed on medium screens
if (isCollapsed) {
return (
<TooltipProvider key={index}>
<Tooltip delayDuration={0}>
<TooltipTrigger asChild>
{button}
</TooltipTrigger>
<TooltipContent side="right" className="hidden md:block xl:hidden">
{link.label}
</TooltipContent>
</Tooltip>
</TooltipProvider>
);
}
return button;
})}
</nav>
<div className="flex-1 min-h-0" />
<div className="px-4 py-4 flex-shrink-0">
<div className="rounded-md bg-muted p-3 lg:p-3">
<h4 className="text-sm font-medium mb-2">Need Help?</h4>
<p className="text-xs text-muted-foreground mb-3 lg:mb-2">
Check out the documentation for help with setup and configuration.
</p>
<a
href="/docs"
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1.5 text-xs lg:text-xs text-primary hover:underline py-2 lg:py-0"
>
Documentation
<ExternalLink className="h-3.5 w-3.5 lg:h-3 lg:w-3" />
</a>
<div className={cn(
"py-4 flex-shrink-0",
isCollapsed ? "md:px-2 xl:px-4 px-4" : "px-4"
)}>
<div className={cn(
"rounded-md bg-muted transition-all duration-200",
isCollapsed ? "md:p-0 xl:p-3 p-3" : "p-3"
)}>
<div className={cn(
isCollapsed ? "md:hidden xl:block" : "block"
)}>
<h4 className="text-sm font-medium mb-2">Need Help?</h4>
<p className="text-xs text-muted-foreground mb-3 md:mb-2">
Check out the documentation for help with setup and configuration.
</p>
<a
href="/docs"
target="_blank"
rel="noopener noreferrer"
className="inline-flex items-center gap-1.5 text-xs md:text-xs text-primary hover:underline py-2 md:py-0"
>
Documentation
<ExternalLink className="h-3.5 w-3.5 md:h-3 md:w-3" />
</a>
</div>
{/* Icon-only help button for collapsed state on medium screens */}
<TooltipProvider>
<Tooltip delayDuration={0}>
<TooltipTrigger asChild>
<a
href="/docs"
target="_blank"
rel="noopener noreferrer"
className={cn(
"flex items-center justify-center rounded-md hover:bg-accent transition-colors",
isCollapsed ? "md:h-12 md:w-12 xl:hidden hidden" : "hidden"
)}
>
<ExternalLink className="h-5 w-5" />
</a>
</TooltipTrigger>
<TooltipContent side="right">
Documentation
</TooltipContent>
</Tooltip>
</TooltipProvider>
</div>
<div className={cn(
isCollapsed ? "md:hidden xl:block" : "block"
)}>
<VersionInfo />
</div>
<VersionInfo />
</div>
</div>
</aside>

View File

@@ -228,17 +228,17 @@ export function OrganizationList({
{(() => {
const parts = [];
if (org.publicRepositoryCount && org.publicRepositoryCount > 0) {
parts.push(`${org.publicRepositoryCount}pub`);
parts.push(`${org.publicRepositoryCount} pub`);
}
if (org.privateRepositoryCount && org.privateRepositoryCount > 0) {
parts.push(`${org.privateRepositoryCount}priv`);
parts.push(`${org.privateRepositoryCount} priv`);
}
if (org.forkRepositoryCount && org.forkRepositoryCount > 0) {
parts.push(`${org.forkRepositoryCount}fork`);
parts.push(`${org.forkRepositoryCount} fork`);
}
return parts.length > 0 ? (
<span className="ml-1">({parts.join('/')})</span>
<span className="ml-1">({parts.join(' | ')})</span>
) : null;
})()}
</div>

View File

@@ -320,7 +320,7 @@ export default function Repository() {
const selectedRepos = repositories.filter(repo => repo.id && selectedRepoIds.has(repo.id));
const eligibleRepos = selectedRepos.filter(
repo => repo.status === "mirrored" || repo.status === "synced"
repo => ["mirrored", "synced", "archived"].includes(repo.status)
);
if (eligibleRepos.length === 0) {

View File

@@ -5,7 +5,7 @@ import { FlipHorizontal, GitFork, RefreshCw, RotateCcw, Star, Lock, Ban, Check,
import { SiGithub, SiGitea } from "react-icons/si";
import type { Repository } from "@/lib/db/schema";
import { Button } from "@/components/ui/button";
import { formatDate, getStatusColor } from "@/lib/utils";
import { formatDate, formatLastSyncTime, getStatusColor } from "@/lib/utils";
import type { FilterParams } from "@/types/filter";
import { Skeleton } from "@/components/ui/skeleton";
import { useGiteaConfig } from "@/hooks/useGiteaConfig";
@@ -90,7 +90,7 @@ export default function RepositoryTable({
}
// Only provide Gitea links for repositories that have been or are being mirrored
const validStatuses = ['mirroring', 'mirrored', 'syncing', 'synced'];
const validStatuses = ['mirroring', 'mirrored', 'syncing', 'synced', 'archived'];
if (!validStatuses.includes(repository.status)) {
return null;
}
@@ -242,7 +242,7 @@ export default function RepositoryTable({
{repo.status}
</Badge>
<span className="text-xs text-muted-foreground">
{repo.lastMirrored ? formatDate(repo.lastMirrored) : "Never mirrored"}
{formatLastSyncTime(repo.lastMirrored)}
</span>
</div>
</div>
@@ -410,7 +410,7 @@ export default function RepositoryTable({
<div className="h-full p-3 flex items-center justify-center flex-[0.3]">
<Skeleton className="h-4 w-4" />
</div>
<div className="h-full p-3 text-sm font-medium flex-[2.5]">
<div className="h-full py-3 text-sm font-medium flex-[2.3]">
Repository
</div>
<div className="h-full p-3 text-sm font-medium flex-[1]">Owner</div>
@@ -437,7 +437,7 @@ export default function RepositoryTable({
<div className="h-full p-3 flex items-center justify-center flex-[0.3]">
<Skeleton className="h-4 w-4" />
</div>
<div className="h-full p-3 flex-[2.5]">
<div className="h-full p-3 flex-[2.3]">
<Skeleton className="h-5 w-48" />
<Skeleton className="h-3 w-24 mt-1" />
</div>
@@ -530,7 +530,7 @@ export default function RepositoryTable({
aria-label="Select all repositories"
/>
</div>
<div className="h-full p-3 text-sm font-medium flex-[2.5]">
<div className="h-full py-3 text-sm font-medium flex-[2.3]">
Repository
</div>
<div className="h-full p-3 text-sm font-medium flex-[1]">Owner</div>
@@ -588,7 +588,7 @@ export default function RepositoryTable({
</div>
{/* Repository */}
<div className="h-full py-3 flex items-center gap-2 flex-[2.5]">
<div className="h-full py-3 flex items-center gap-2 flex-[2.3]">
<div className="flex-1">
<div className="font-medium flex items-center gap-1">
{repo.name}
@@ -629,9 +629,7 @@ export default function RepositoryTable({
{/* Last Mirrored */}
<div className="h-full p-3 flex items-center flex-[1]">
<p className="text-sm">
{repo.lastMirrored
? formatDate(new Date(repo.lastMirrored))
: "Never"}
{formatLastSyncTime(repo.lastMirrored)}
</p>
</div>
@@ -822,8 +820,8 @@ function RepoActionButton({
primaryLabel = "Retry";
primaryIcon = <RotateCcw className="h-4 w-4" />;
primaryOnClick = onRetry;
} else if (["mirrored", "synced", "syncing"].includes(repo.status)) {
primaryLabel = "Sync";
} else if (["mirrored", "synced", "syncing", "archived"].includes(repo.status)) {
primaryLabel = repo.status === "archived" ? "Manual Sync" : "Sync";
primaryIcon = <RefreshCw className="h-4 w-4" />;
primaryOnClick = onSync;
primaryDisabled ||= repo.status === "syncing";
@@ -891,4 +889,4 @@ function RepoActionButton({
</DropdownMenuContent>
</DropdownMenu>
);
}
}

View File

@@ -0,0 +1,30 @@
import * as React from "react"
import * as ProgressPrimitive from "@radix-ui/react-progress"
import { cn } from "@/lib/utils"
interface ProgressProps extends React.ComponentPropsWithoutRef<typeof ProgressPrimitive.Root> {
indicatorClassName?: string
}
const Progress = React.forwardRef<
React.ElementRef<typeof ProgressPrimitive.Root>,
ProgressProps
>(({ className, value, indicatorClassName, ...props }, ref) => (
<ProgressPrimitive.Root
ref={ref}
className={cn(
"relative h-4 w-full overflow-hidden rounded-full bg-secondary",
className
)}
{...props}
>
<ProgressPrimitive.Indicator
className={cn("h-full w-full flex-1 bg-primary transition-all", indicatorClassName)}
style={{ transform: `translateX(-${100 - (value || 0)}%)` }}
/>
</ProgressPrimitive.Root>
))
Progress.displayName = ProgressPrimitive.Root.displayName
export { Progress }

View File

@@ -1,12 +1,39 @@
import "@/lib/polyfills/buffer";
import { createAuthClient } from "better-auth/react";
import { oidcClient } from "better-auth/client/plugins";
import { ssoClient } from "@better-auth/sso/client";
import type { Session as BetterAuthSession, User as BetterAuthUser } from "better-auth";
export const authClient = createAuthClient({
// The base URL is optional when running on the same domain
// Better Auth will use the current domain by default
baseURL: typeof window !== 'undefined' ? window.location.origin : 'http://localhost:4321',
// Use PUBLIC_BETTER_AUTH_URL if set (for multi-origin access), otherwise use current origin
// This allows the client to connect to the auth server even when accessed from different origins
baseURL: (() => {
let url: string | undefined;
// Check for public environment variable first (for client-side access)
if (typeof import.meta !== 'undefined' && import.meta.env?.PUBLIC_BETTER_AUTH_URL) {
url = import.meta.env.PUBLIC_BETTER_AUTH_URL;
}
// Validate and clean the URL if provided
if (url && typeof url === 'string' && url.trim() !== '') {
try {
// Validate URL format and remove trailing slash
const validatedUrl = new URL(url.trim());
return validatedUrl.origin; // Use origin to ensure clean URL without path
} catch (e) {
console.warn(`Invalid PUBLIC_BETTER_AUTH_URL: ${url}, falling back to default`);
}
}
// Fall back to current origin if running in browser
if (typeof window !== 'undefined' && window.location?.origin) {
return window.location.origin;
}
// Default for SSR - always return a valid URL
return 'http://localhost:4321';
})(),
basePath: '/api/auth', // Explicitly set the base path
plugins: [
oidcClient(),
@@ -34,4 +61,4 @@ export type Session = BetterAuthSession & {
};
export type AuthUser = BetterAuthUser & {
username?: string | null;
};
};

View File

@@ -19,42 +19,71 @@ export const auth = betterAuth({
// Base URL configuration - use the primary URL (Better Auth only supports single baseURL)
baseURL: (() => {
const url = process.env.BETTER_AUTH_URL || "http://localhost:4321";
const url = process.env.BETTER_AUTH_URL;
const defaultUrl = "http://localhost:4321";
// Check if URL is provided and not empty
if (!url || typeof url !== 'string' || url.trim() === '') {
console.info('BETTER_AUTH_URL not set, using default:', defaultUrl);
return defaultUrl;
}
try {
// Validate URL format
new URL(url);
return url;
} catch {
console.warn(`Invalid BETTER_AUTH_URL: ${url}, falling back to localhost`);
return "http://localhost:4321";
// Validate URL format and ensure it's a proper origin
const validatedUrl = new URL(url.trim());
const cleanUrl = validatedUrl.origin; // Use origin to ensure no trailing paths
console.info('Using BETTER_AUTH_URL:', cleanUrl);
return cleanUrl;
} catch (e) {
console.error(`Invalid BETTER_AUTH_URL format: "${url}"`);
console.error('Error:', e);
console.info('Falling back to default:', defaultUrl);
return defaultUrl;
}
})(),
basePath: "/api/auth", // Specify the base path for auth endpoints
// Trusted origins - this is how we support multiple access URLs
trustedOrigins: (() => {
const origins = [
const origins: string[] = [
"http://localhost:4321",
"http://localhost:8080", // Keycloak
];
// Add the primary URL from BETTER_AUTH_URL
const primaryUrl = process.env.BETTER_AUTH_URL || "http://localhost:4321";
try {
new URL(primaryUrl);
origins.push(primaryUrl);
} catch {
// Skip if invalid
const primaryUrl = process.env.BETTER_AUTH_URL;
if (primaryUrl && typeof primaryUrl === 'string' && primaryUrl.trim() !== '') {
try {
const validatedUrl = new URL(primaryUrl.trim());
origins.push(validatedUrl.origin);
} catch {
// Skip if invalid
}
}
// Add additional trusted origins from environment
// This is where users can specify multiple access URLs
if (process.env.BETTER_AUTH_TRUSTED_ORIGINS) {
origins.push(...process.env.BETTER_AUTH_TRUSTED_ORIGINS.split(',').map(o => o.trim()));
const additionalOrigins = process.env.BETTER_AUTH_TRUSTED_ORIGINS
.split(',')
.map(o => o.trim())
.filter(o => o !== '');
// Validate each additional origin
for (const origin of additionalOrigins) {
try {
const validatedUrl = new URL(origin);
origins.push(validatedUrl.origin);
} catch {
console.warn(`Invalid trusted origin: ${origin}, skipping`);
}
}
}
// Remove duplicates and return
return [...new Set(origins.filter(Boolean))];
// Remove duplicates and empty strings, then return
const uniqueOrigins = [...new Set(origins.filter(Boolean))];
console.info('Trusted origins:', uniqueOrigins);
return uniqueOrigins;
})(),
// Authentication methods
@@ -142,4 +171,4 @@ export const auth = betterAuth({
});
// Export type for use in other parts of the app
export type Auth = typeof auth;
export type Auth = typeof auth;

View File

@@ -82,5 +82,6 @@ export {
oauthApplications,
oauthAccessTokens,
oauthConsent,
ssoProviders
ssoProviders,
rateLimits
} from "./schema";

View File

@@ -1,5 +1,5 @@
import { z } from "zod";
import { sqliteTable, text, integer, index } from "drizzle-orm/sqlite-core";
import { sqliteTable, text, integer, index, uniqueIndex } from "drizzle-orm/sqlite-core";
import { sql } from "drizzle-orm";
// ===== Zod Validation Schemas =====
@@ -27,7 +27,9 @@ export const githubConfigSchema = z.object({
starredReposOrg: z.string().optional(),
mirrorStrategy: z.enum(["preserve", "single-org", "flat-user", "mixed"]).default("preserve"),
defaultOrg: z.string().optional(),
skipStarredIssues: z.boolean().default(false),
starredCodeOnly: z.boolean().default(false),
skipStarredIssues: z.boolean().optional(), // Deprecated: kept for backward compatibility, use starredCodeOnly instead
starredDuplicateStrategy: z.enum(["suffix", "prefix", "owner-org"]).default("suffix").optional(),
});
export const giteaConfigSchema = z.object({
@@ -80,6 +82,8 @@ export const scheduleConfigSchema = z.object({
updateInterval: z.number().default(86400000),
skipRecentlyMirrored: z.boolean().default(true),
recentThreshold: z.number().default(3600000),
autoImport: z.boolean().default(true),
autoMirror: z.boolean().default(false),
lastRun: z.coerce.date().optional(),
nextRun: z.coerce.date().optional(),
});
@@ -90,7 +94,7 @@ export const cleanupConfigSchema = z.object({
deleteFromGitea: z.boolean().default(false),
deleteIfNotInGitHub: z.boolean().default(true),
protectedRepos: z.array(z.string()).default([]),
dryRun: z.boolean().default(true),
dryRun: z.boolean().default(false),
orphanedRepoAction: z
.enum(["skip", "archive", "delete"])
.default("archive"),
@@ -151,6 +155,7 @@ export const repositorySchema = z.object({
"deleted",
"syncing",
"synced",
"archived",
])
.default("imported"),
lastMirrored: z.coerce.date().optional().nullable(),
@@ -180,6 +185,7 @@ export const mirrorJobSchema = z.object({
"deleted",
"syncing",
"synced",
"archived",
])
.default("imported"),
message: z.string(),
@@ -202,7 +208,7 @@ export const organizationSchema = z.object({
configId: z.string(),
name: z.string(),
avatarUrl: z.string(),
membershipRole: z.enum(["admin", "member", "owner"]).default("member"),
membershipRole: z.enum(["member", "admin", "owner", "billing_manager"]).default("member"),
isIncluded: z.boolean().default(true),
destinationOrg: z.string().optional().nullable(),
status: z
@@ -379,6 +385,7 @@ export const repositories = sqliteTable("repositories", {
index("idx_repositories_organization").on(table.organization),
index("idx_repositories_is_fork").on(table.isForked),
index("idx_repositories_is_starred").on(table.isStarred),
uniqueIndex("uniq_repositories_user_full_name").on(table.userId, table.fullName),
]);
export const mirrorJobs = sqliteTable("mirror_jobs", {
@@ -445,6 +452,9 @@ export const organizations = sqliteTable("organizations", {
errorMessage: text("error_message"),
repositoryCount: integer("repository_count").notNull().default(0),
publicRepositoryCount: integer("public_repository_count"),
privateRepositoryCount: integer("private_repository_count"),
forkRepositoryCount: integer("fork_repository_count"),
createdAt: integer("created_at", { mode: "timestamp" })
.notNull()
@@ -623,10 +633,52 @@ export const ssoProviders = sqliteTable("sso_providers", {
index("idx_sso_providers_issuer").on(table.issuer),
]);
// ===== Rate Limit Tracking =====
export const rateLimitSchema = z.object({
id: z.string(),
userId: z.string(),
provider: z.enum(["github", "gitea"]).default("github"),
limit: z.number(),
remaining: z.number(),
used: z.number(),
reset: z.coerce.date(),
retryAfter: z.number().optional(), // seconds to wait
status: z.enum(["ok", "warning", "limited", "exceeded"]).default("ok"),
lastChecked: z.coerce.date(),
createdAt: z.coerce.date(),
updatedAt: z.coerce.date(),
});
export const rateLimits = sqliteTable("rate_limits", {
id: text("id").primaryKey(),
userId: text("user_id")
.notNull()
.references(() => users.id),
provider: text("provider").notNull().default("github"),
limit: integer("limit").notNull(),
remaining: integer("remaining").notNull(),
used: integer("used").notNull(),
reset: integer("reset", { mode: "timestamp" }).notNull(),
retryAfter: integer("retry_after"), // seconds to wait
status: text("status").notNull().default("ok"),
lastChecked: integer("last_checked", { mode: "timestamp" }).notNull(),
createdAt: integer("created_at", { mode: "timestamp" })
.notNull()
.default(sql`(unixepoch())`),
updatedAt: integer("updated_at", { mode: "timestamp" })
.notNull()
.default(sql`(unixepoch())`),
}, (table) => [
index("idx_rate_limits_user_provider").on(table.userId, table.provider),
index("idx_rate_limits_status").on(table.status),
]);
// Export type definitions
export type User = z.infer<typeof userSchema>;
export type Config = z.infer<typeof configSchema>;
export type Repository = z.infer<typeof repositorySchema>;
export type MirrorJob = z.infer<typeof mirrorJobSchema>;
export type Organization = z.infer<typeof organizationSchema>;
export type Event = z.infer<typeof eventSchema>;
export type Event = z.infer<typeof eventSchema>;
export type RateLimit = z.infer<typeof rateLimitSchema>;

View File

@@ -21,7 +21,7 @@ interface EnvConfig {
mirrorOrganizations?: boolean;
preserveOrgStructure?: boolean;
onlyMirrorOrgs?: boolean;
skipStarredIssues?: boolean;
starredCodeOnly?: boolean;
starredReposOrg?: string;
mirrorStrategy?: 'preserve' | 'single-org' | 'flat-user' | 'mixed';
};
@@ -69,6 +69,8 @@ interface EnvConfig {
updateInterval?: number;
skipRecentlyMirrored?: boolean;
recentThreshold?: number;
autoImport?: boolean;
autoMirror?: boolean;
};
cleanup: {
enabled?: boolean;
@@ -105,7 +107,7 @@ function parseEnvConfig(): EnvConfig {
mirrorOrganizations: process.env.MIRROR_ORGANIZATIONS === 'true',
preserveOrgStructure: process.env.PRESERVE_ORG_STRUCTURE === 'true',
onlyMirrorOrgs: process.env.ONLY_MIRROR_ORGS === 'true',
skipStarredIssues: process.env.SKIP_STARRED_ISSUES === 'true',
starredCodeOnly: process.env.SKIP_STARRED_ISSUES === 'true',
starredReposOrg: process.env.STARRED_REPOS_ORG,
mirrorStrategy: process.env.MIRROR_STRATEGY as 'preserve' | 'single-org' | 'flat-user' | 'mixed',
},
@@ -133,6 +135,7 @@ function parseEnvConfig(): EnvConfig {
mirrorLabels: process.env.MIRROR_LABELS === 'true',
mirrorMilestones: process.env.MIRROR_MILESTONES === 'true',
mirrorMetadata: process.env.MIRROR_METADATA === 'true',
releaseLimit: process.env.RELEASE_LIMIT ? parseInt(process.env.RELEASE_LIMIT, 10) : undefined,
},
schedule: {
enabled: process.env.SCHEDULE_ENABLED === 'true' ||
@@ -156,6 +159,8 @@ function parseEnvConfig(): EnvConfig {
updateInterval: process.env.SCHEDULE_UPDATE_INTERVAL ? parseInt(process.env.SCHEDULE_UPDATE_INTERVAL, 10) : undefined,
skipRecentlyMirrored: process.env.SCHEDULE_SKIP_RECENTLY_MIRRORED === 'true',
recentThreshold: process.env.SCHEDULE_RECENT_THRESHOLD ? parseInt(process.env.SCHEDULE_RECENT_THRESHOLD, 10) : undefined,
autoImport: process.env.AUTO_IMPORT_REPOS !== 'false',
autoMirror: process.env.AUTO_MIRROR_REPOS === 'true',
},
cleanup: {
enabled: process.env.CLEANUP_ENABLED === 'true' ||
@@ -164,7 +169,7 @@ function parseEnvConfig(): EnvConfig {
deleteFromGitea: process.env.CLEANUP_DELETE_FROM_GITEA === 'true',
deleteIfNotInGitHub: process.env.CLEANUP_DELETE_IF_NOT_IN_GITHUB === 'true',
protectedRepos,
dryRun: process.env.CLEANUP_DRY_RUN === 'true',
dryRun: process.env.CLEANUP_DRY_RUN === 'true' ? true : process.env.CLEANUP_DRY_RUN === 'false' ? false : false,
orphanedRepoAction: process.env.CLEANUP_ORPHANED_REPO_ACTION as 'skip' | 'archive' | 'delete',
batchSize: process.env.CLEANUP_BATCH_SIZE ? parseInt(process.env.CLEANUP_BATCH_SIZE, 10) : undefined,
pauseBetweenDeletes: process.env.CLEANUP_PAUSE_BETWEEN_DELETES ? parseInt(process.env.CLEANUP_PAUSE_BETWEEN_DELETES, 10) : undefined,
@@ -248,7 +253,7 @@ export async function initializeConfigFromEnv(): Promise<void> {
starredReposOrg: envConfig.github.starredReposOrg || existingConfig?.[0]?.githubConfig?.starredReposOrg || 'starred',
mirrorStrategy,
defaultOrg: envConfig.gitea.organization || existingConfig?.[0]?.githubConfig?.defaultOrg || 'github-mirrors',
skipStarredIssues: envConfig.github.skipStarredIssues ?? existingConfig?.[0]?.githubConfig?.skipStarredIssues ?? false,
starredCodeOnly: envConfig.github.starredCodeOnly ?? existingConfig?.[0]?.githubConfig?.starredCodeOnly ?? false,
};
// Build Gitea config
@@ -271,6 +276,7 @@ export async function initializeConfigFromEnv(): Promise<void> {
forkStrategy: envConfig.gitea.forkStrategy || existingConfig?.[0]?.giteaConfig?.forkStrategy || 'reference',
// Mirror metadata options
mirrorReleases: envConfig.mirror.mirrorReleases ?? existingConfig?.[0]?.giteaConfig?.mirrorReleases ?? false,
releaseLimit: envConfig.mirror.releaseLimit ?? existingConfig?.[0]?.giteaConfig?.releaseLimit ?? 10,
mirrorMetadata: envConfig.mirror.mirrorMetadata ?? (envConfig.mirror.mirrorIssues || envConfig.mirror.mirrorPullRequests || envConfig.mirror.mirrorLabels || envConfig.mirror.mirrorMilestones) ?? existingConfig?.[0]?.giteaConfig?.mirrorMetadata ?? false,
mirrorIssues: envConfig.mirror.mirrorIssues ?? existingConfig?.[0]?.giteaConfig?.mirrorIssues ?? false,
mirrorPullRequests: envConfig.mirror.mirrorPullRequests ?? existingConfig?.[0]?.giteaConfig?.mirrorPullRequests ?? false,
@@ -299,6 +305,8 @@ export async function initializeConfigFromEnv(): Promise<void> {
updateInterval: envConfig.schedule.updateInterval ?? existingConfig?.[0]?.scheduleConfig?.updateInterval ?? 86400000,
skipRecentlyMirrored: envConfig.schedule.skipRecentlyMirrored ?? existingConfig?.[0]?.scheduleConfig?.skipRecentlyMirrored ?? true,
recentThreshold: envConfig.schedule.recentThreshold ?? existingConfig?.[0]?.scheduleConfig?.recentThreshold ?? 3600000,
autoImport: envConfig.schedule.autoImport ?? existingConfig?.[0]?.scheduleConfig?.autoImport ?? true,
autoMirror: envConfig.schedule.autoMirror ?? existingConfig?.[0]?.scheduleConfig?.autoMirror ?? false,
lastRun: existingConfig?.[0]?.scheduleConfig?.lastRun || undefined,
nextRun: existingConfig?.[0]?.scheduleConfig?.nextRun || undefined,
};
@@ -356,4 +364,4 @@ export async function initializeConfigFromEnv(): Promise<void> {
console.error('[ENV Config Loader] Failed to initialize configuration from environment:', error);
// Don't throw - this is a non-critical initialization
}
}
}

View File

@@ -7,6 +7,9 @@ mock.module("@/lib/helpers", () => ({
createMirrorJob: mockCreateMirrorJob
}));
const mockMirrorGitHubReleasesToGitea = mock(() => Promise.resolve());
const mockGetGiteaRepoOwnerAsync = mock(() => Promise.resolve("starred"));
// Mock the database module
const mockDb = {
insert: mock((table: any) => ({
@@ -220,6 +223,9 @@ describe("Enhanced Gitea Operations", () => {
mockCreateMirrorJob.mockClear();
mockDb.insert.mockClear();
mockDb.update.mockClear();
mockMirrorGitHubReleasesToGitea.mockClear();
mockGetGiteaRepoOwnerAsync.mockClear();
mockGetGiteaRepoOwnerAsync.mockImplementation(() => Promise.resolve("starred"));
// Reset tracking variables
orgCheckCount = 0;
orgTestContext = "";
@@ -250,6 +256,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -280,6 +287,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -306,6 +314,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -362,6 +371,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -391,6 +401,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -409,18 +420,17 @@ describe("Enhanced Gitea Operations", () => {
updatedAt: new Date(),
};
// Mock getGiteaRepoOwnerAsync
const mockGetOwner = mock(() => Promise.resolve("starred"));
global.import = mock(async (path: string) => {
if (path === "./gitea") {
return { getGiteaRepoOwnerAsync: mockGetOwner };
}
return {};
}) as any;
await expect(
syncGiteaRepoEnhanced({ config, repository })
syncGiteaRepoEnhanced(
{ config, repository },
{
getGiteaRepoOwnerAsync: mockGetGiteaRepoOwnerAsync,
mirrorGitHubReleasesToGitea: mockMirrorGitHubReleasesToGitea,
}
)
).rejects.toThrow("Repository non-mirror-repo is not a mirror. Cannot sync.");
expect(mockMirrorGitHubReleasesToGitea).not.toHaveBeenCalled();
});
test("should successfully sync a mirror repository", async () => {
@@ -436,6 +446,7 @@ describe("Enhanced Gitea Operations", () => {
url: "https://gitea.example.com",
token: "encrypted-token",
defaultOwner: "testuser",
mirrorReleases: true,
},
};
@@ -454,18 +465,22 @@ describe("Enhanced Gitea Operations", () => {
updatedAt: new Date(),
};
// Mock getGiteaRepoOwnerAsync
const mockGetOwner = mock(() => Promise.resolve("starred"));
global.import = mock(async (path: string) => {
if (path === "./gitea") {
return { getGiteaRepoOwnerAsync: mockGetOwner };
const result = await syncGiteaRepoEnhanced(
{ config, repository },
{
getGiteaRepoOwnerAsync: mockGetGiteaRepoOwnerAsync,
mirrorGitHubReleasesToGitea: mockMirrorGitHubReleasesToGitea,
}
return {};
}) as any;
const result = await syncGiteaRepoEnhanced({ config, repository });
);
expect(result).toEqual({ success: true });
expect(mockGetGiteaRepoOwnerAsync).toHaveBeenCalled();
expect(mockMirrorGitHubReleasesToGitea).toHaveBeenCalledTimes(1);
const releaseCall = mockMirrorGitHubReleasesToGitea.mock.calls[0][0];
expect(releaseCall.giteaOwner).toBe("starred");
expect(releaseCall.giteaRepoName).toBe("mirror-repo");
expect(releaseCall.config.githubConfig?.token).toBe("github-token");
expect(releaseCall.octokit).toBeDefined();
});
});
@@ -567,4 +582,4 @@ describe("Enhanced Gitea Operations", () => {
expect(deleteCalled).toBe(true);
});
});
});
});

View File

@@ -8,13 +8,19 @@
import type { Config } from "@/types/config";
import type { Repository } from "./db/schema";
import { Octokit } from "@octokit/rest";
import { createMirrorJob } from "./helpers";
import { decryptConfigTokens } from "./utils/config-encryption";
import { httpPost, httpGet, HttpError } from "./http-client";
import { httpPost, httpGet, httpPatch, HttpError } from "./http-client";
import { db, repositories } from "./db";
import { eq } from "drizzle-orm";
import { repoStatusEnum } from "@/types/Repository";
type SyncDependencies = {
getGiteaRepoOwnerAsync: typeof import("./gitea")["getGiteaRepoOwnerAsync"];
mirrorGitHubReleasesToGitea: typeof import("./gitea")["mirrorGitHubReleasesToGitea"];
};
/**
* Enhanced repository information including mirror status
*/
@@ -239,7 +245,7 @@ export async function syncGiteaRepoEnhanced({
}: {
config: Partial<Config>;
repository: Repository;
}): Promise<any> {
}, deps?: SyncDependencies): Promise<any> {
try {
if (!config.userId || !config.giteaConfig?.url || !config.giteaConfig?.token) {
throw new Error("Gitea config is required.");
@@ -259,8 +265,8 @@ export async function syncGiteaRepoEnhanced({
.where(eq(repositories.id, repository.id!));
// Get the expected owner
const { getGiteaRepoOwnerAsync } = await import("./gitea");
const repoOwner = await getGiteaRepoOwnerAsync({ config, repository });
const dependencies = deps ?? (await import("./gitea"));
const repoOwner = await dependencies.getGiteaRepoOwnerAsync({ config, repository });
// Check if repo exists and get its info
const repoInfo = await getGiteaRepoInfo({
@@ -299,6 +305,23 @@ export async function syncGiteaRepoEnhanced({
throw new Error(`Repository ${repository.name} is not a mirror. Cannot sync.`);
}
// Update mirror interval if needed
if (config.giteaConfig?.mirrorInterval) {
try {
console.log(`[Sync] Updating mirror interval for ${repository.name} to ${config.giteaConfig.mirrorInterval}`);
const updateUrl = `${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repository.name}`;
await httpPatch(updateUrl, {
mirror_interval: config.giteaConfig.mirrorInterval,
}, {
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
});
console.log(`[Sync] Successfully updated mirror interval for ${repository.name}`);
} catch (updateError) {
console.warn(`[Sync] Failed to update mirror interval for ${repository.name}:`, updateError);
// Continue with sync even if interval update fails
}
}
// Perform the sync
const apiUrl = `${config.giteaConfig.url}/api/v1/repos/${repoOwner}/${repository.name}/mirror-sync`;
@@ -307,6 +330,36 @@ export async function syncGiteaRepoEnhanced({
Authorization: `token ${decryptedConfig.giteaConfig.token}`,
});
const shouldMirrorReleases =
decryptedConfig.giteaConfig?.mirrorReleases &&
!(repository.isStarred && decryptedConfig.githubConfig?.starredCodeOnly);
if (shouldMirrorReleases) {
if (!decryptedConfig.githubConfig?.token) {
console.warn(
`[Sync] Skipping release mirroring for ${repository.name}: Missing GitHub token`
);
} else {
try {
const octokit = new Octokit({ auth: decryptedConfig.githubConfig.token });
await dependencies.mirrorGitHubReleasesToGitea({
config: decryptedConfig,
octokit,
repository,
giteaOwner: repoOwner,
giteaRepoName: repository.name,
});
console.log(`[Sync] Mirrored releases for ${repository.name} after sync`);
} catch (releaseError) {
console.error(
`[Sync] Failed to mirror releases for ${repository.name}: ${
releaseError instanceof Error ? releaseError.message : String(releaseError)
}`
);
}
}
}
// Mark repo as "synced" in DB
await db
.update(repositories)
@@ -518,4 +571,4 @@ export async function handleExistingNonMirrorRepo({
// TODO: Implement rename strategy if needed
break;
}
}
}

View File

@@ -8,10 +8,19 @@ import { createMockResponse, mockFetch } from "@/tests/mock-fetch";
// Mock the isRepoPresentInGitea function
const mockIsRepoPresentInGitea = mock(() => Promise.resolve(false));
let mockDbSelectResult: any[] = [];
// Mock the database module
mock.module("@/lib/db", () => {
return {
db: {
select: () => ({
from: () => ({
where: () => ({
limit: () => Promise.resolve(mockDbSelectResult)
})
})
}),
update: () => ({
set: () => ({
where: () => Promise.resolve()
@@ -63,6 +72,7 @@ describe("Gitea Repository Mirroring", () => {
originalConsoleError = console.error;
console.log = mock(() => {});
console.error = mock(() => {});
mockDbSelectResult = [];
});
afterEach(() => {
@@ -331,7 +341,7 @@ describe("getGiteaRepoOwner - Organization Override Tests", () => {
excludeOrgs: [],
mirrorPublicOrgs: false,
publicOrgs: [],
skipStarredIssues: false,
starredCodeOnly: false,
mirrorStrategy: "preserve"
},
giteaConfig: {
@@ -449,4 +459,37 @@ describe("getGiteaRepoOwner - Organization Override Tests", () => {
const result = getGiteaRepoOwner({ config: configWithFlatUser, repository: repo });
expect(result).toBe("giteauser");
});
test("getGiteaRepoOwnerAsync honors organization override for owner role", async () => {
mockDbSelectResult = [
{
id: "org-id",
userId: "user-id",
configId: "config-id",
name: "myorg",
membershipRole: "owner",
status: "imported",
destinationOrg: "custom-org",
avatarUrl: "https://example.com/avatar.png",
isIncluded: true,
repositoryCount: 0,
createdAt: new Date(),
updatedAt: new Date()
}
];
const configWithUser: Partial<Config> = {
...baseConfig,
userId: "user-id"
};
const repo = { ...baseRepo, organization: "myorg" };
const result = await getGiteaRepoOwnerAsync({
config: configWithUser,
repository: repo
});
expect(result).toBe("custom-org");
});
});

File diff suppressed because it is too large Load Diff

View File

@@ -1,15 +1,179 @@
import type { GitOrg, MembershipRole } from "@/types/organizations";
import type { GitRepo, RepoStatus } from "@/types/Repository";
import { Octokit } from "@octokit/rest";
import { throttling } from "@octokit/plugin-throttling";
import type { Config } from "@/types/config";
// Conditionally import rate limit manager (not available in test environment)
let RateLimitManager: any = null;
let publishEvent: any = null;
if (process.env.NODE_ENV !== "test") {
try {
const rateLimitModule = await import("@/lib/rate-limit-manager");
RateLimitManager = rateLimitModule.RateLimitManager;
const eventsModule = await import("@/lib/events");
publishEvent = eventsModule.publishEvent;
} catch (error) {
console.warn("Rate limit manager not available:", error);
}
}
// Extend Octokit with throttling plugin when available (tests may stub Octokit)
// Fallback to base Octokit if .plugin is not present
const MyOctokit: any = (Octokit as any)?.plugin?.call
? (Octokit as any).plugin(throttling)
: Octokit as any;
/**
* Creates an authenticated Octokit instance
* Creates an authenticated Octokit instance with rate limit tracking and throttling
*/
export function createGitHubClient(token: string): Octokit {
return new Octokit({
auth: token,
export function createGitHubClient(token: string, userId?: string, username?: string): Octokit {
// Create a proper User-Agent to identify our application
// This helps GitHub understand our traffic patterns and can provide better rate limits
const userAgent = username
? `gitea-mirror/3.5.4 (user:${username})`
: "gitea-mirror/3.5.4";
const octokit = new MyOctokit({
auth: token, // Always use token for authentication (5000 req/hr vs 60 for unauthenticated)
userAgent, // Identify our application and user
baseUrl: "https://api.github.com", // Explicitly set the API endpoint
log: {
debug: () => {},
info: console.log,
warn: console.warn,
error: console.error,
},
request: {
// Add default headers for better identification
headers: {
accept: "application/vnd.github.v3+json",
"x-github-api-version": "2022-11-28", // Use a stable API version
},
},
throttle: {
onRateLimit: async (retryAfter: number, options: any, octokit: any, retryCount: number) => {
const isSearch = options.url.includes("/search/");
const maxRetries = isSearch ? 5 : 3; // Search endpoints get more retries
console.warn(
`[GitHub] Rate limit hit for ${options.method} ${options.url}. Retry ${retryCount + 1}/${maxRetries}`
);
// Update rate limit status and notify UI (if available)
if (userId && RateLimitManager) {
await RateLimitManager.updateFromResponse(userId, {
"retry-after": retryAfter.toString(),
"x-ratelimit-remaining": "0",
"x-ratelimit-reset": (Date.now() / 1000 + retryAfter).toString(),
});
}
if (userId && publishEvent) {
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "rate-limited",
provider: "github",
retryAfter,
retryCount,
endpoint: options.url,
message: `Rate limit hit. Waiting ${retryAfter}s before retry ${retryCount + 1}/${maxRetries}...`,
},
});
}
// Retry with exponential backoff
if (retryCount < maxRetries) {
console.log(`[GitHub] Waiting ${retryAfter}s before retry...`);
return true;
}
// Max retries reached
console.error(`[GitHub] Max retries (${maxRetries}) reached for ${options.url}`);
return false;
},
onSecondaryRateLimit: async (retryAfter: number, options: any, octokit: any, retryCount: number) => {
console.warn(
`[GitHub] Secondary rate limit hit for ${options.method} ${options.url}`
);
// Update status and notify UI (if available)
if (userId && publishEvent) {
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "secondary-limited",
provider: "github",
retryAfter,
retryCount,
endpoint: options.url,
message: `Secondary rate limit hit. Waiting ${retryAfter}s...`,
},
});
}
// Retry up to 2 times for secondary rate limits
if (retryCount < 2) {
console.log(`[GitHub] Waiting ${retryAfter}s for secondary rate limit...`);
return true;
}
return false;
},
// Throttle options to prevent hitting limits
fallbackSecondaryRateRetryAfter: 60, // Wait 60s on secondary rate limit
minimumSecondaryRateRetryAfter: 5, // Min 5s wait
retryAfterBaseValue: 1000, // Base retry in ms
},
});
// Add additional rate limit tracking if userId is provided and RateLimitManager is available
if (userId && RateLimitManager) {
octokit.hook.after("request", async (response: any, options: any) => {
// Update rate limit from response headers
if (response.headers) {
await RateLimitManager.updateFromResponse(userId, response.headers);
}
});
octokit.hook.error("request", async (error: any, options: any) => {
// Handle rate limit errors
if (error.status === 403 || error.status === 429) {
const message = error.message || "";
if (message.includes("rate limit") || message.includes("API rate limit")) {
console.error(`[GitHub] Rate limit error for user ${userId}: ${message}`);
// Update rate limit status from error response (if available)
if (error.response?.headers && RateLimitManager) {
await RateLimitManager.updateFromResponse(userId, error.response.headers);
}
// Create error event for UI (if available)
if (publishEvent) {
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "error",
provider: "github",
error: message,
endpoint: options.url,
message: `Rate limit exceeded: ${message}`,
},
});
}
}
}
throw error;
});
}
return octokit;
}
/**
@@ -68,6 +232,8 @@ export async function getGithubRepositories({
owner: repo.owner.login,
organization:
repo.owner.type === "Organization" ? repo.owner.login : undefined,
mirroredLocation: "",
destinationOrg: null,
isPrivate: repo.private,
isForked: repo.fork,
@@ -82,6 +248,8 @@ export async function getGithubRepositories({
hasLFS: false,
hasSubmodules: false,
language: repo.language,
description: repo.description,
defaultBranch: repo.default_branch,
visibility: (repo.visibility ?? "public") as GitRepo["visibility"],
@@ -125,6 +293,8 @@ export async function getGithubStarredRepositories({
owner: repo.owner.login,
organization:
repo.owner.type === "Organization" ? repo.owner.login : undefined,
mirroredLocation: "",
destinationOrg: null,
isPrivate: repo.private,
isForked: repo.fork,
@@ -138,6 +308,8 @@ export async function getGithubStarredRepositories({
hasLFS: false, // Placeholder
hasSubmodules: false, // Placeholder
language: repo.language,
description: repo.description,
defaultBranch: repo.default_branch,
visibility: (repo.visibility ?? "public") as GitRepo["visibility"],
@@ -244,6 +416,8 @@ export async function getGithubOrganizationRepositories({
owner: repo.owner.login,
organization: repo.owner.login,
mirroredLocation: "",
destinationOrg: null,
isPrivate: repo.private,
isForked: repo.fork,
@@ -258,6 +432,8 @@ export async function getGithubOrganizationRepositories({
hasLFS: false,
hasSubmodules: false,
language: repo.language,
description: repo.description,
defaultBranch: repo.default_branch ?? "main",
visibility: (repo.visibility ?? "public") as GitRepo["visibility"],

View File

@@ -178,6 +178,21 @@ export async function httpPut<T = any>(
});
}
/**
* PATCH request
*/
export async function httpPatch<T = any>(
url: string,
body?: any,
headers?: Record<string, string>
): Promise<HttpResponse<T>> {
return httpRequest<T>(url, {
method: 'PATCH',
headers,
body: body ? JSON.stringify(body) : undefined,
});
}
/**
* DELETE request
*/
@@ -220,6 +235,10 @@ export class GiteaHttpClient {
return httpPut<T>(`${this.baseUrl}${endpoint}`, body, this.getHeaders());
}
async patch<T = any>(endpoint: string, body?: any): Promise<HttpResponse<T>> {
return httpPatch<T>(`${this.baseUrl}${endpoint}`, body, this.getHeaders());
}
async delete<T = any>(endpoint: string): Promise<HttpResponse<T>> {
return httpDelete<T>(`${this.baseUrl}${endpoint}`, this.getHeaders());
}

View File

@@ -0,0 +1,5 @@
import { Buffer } from "buffer";
if (typeof globalThis !== "undefined" && (globalThis as any).Buffer === undefined) {
(globalThis as any).Buffer = Buffer;
}

View File

@@ -0,0 +1,422 @@
import { db, rateLimits } from "@/lib/db";
import { eq, and } from "drizzle-orm";
import { v4 as uuidv4 } from "uuid";
import type { Octokit } from "@octokit/rest";
import { publishEvent } from "@/lib/events";
type RateLimitStatus = "ok" | "warning" | "limited" | "exceeded";
interface RateLimitInfo {
limit: number;
remaining: number;
used: number;
reset: Date;
retryAfter?: number;
status: RateLimitStatus;
}
interface RateLimitHeaders {
"x-ratelimit-limit"?: string;
"x-ratelimit-remaining"?: string;
"x-ratelimit-used"?: string;
"x-ratelimit-reset"?: string;
"retry-after"?: string;
}
/**
* Rate limit manager for GitHub API
*
* GitHub API Limits for authenticated users:
* - Primary: 5,000 requests per hour
* - Secondary: 900 points per minute (GET = 1 point, mutations = more)
* - Concurrent: Maximum 100 concurrent requests (recommended: 5-20)
*
* For repositories with many issues/PRs:
* - Each issue = 1 request to fetch
* - Each PR = 1 request to fetch
* - Comments = Additional requests per issue/PR
* - Better to limit by total requests rather than repositories
*/
export class RateLimitManager {
private static readonly WARNING_THRESHOLD = 0.2; // Warn when 20% remaining (80% used)
private static readonly PAUSE_THRESHOLD = 0.05; // Pause when 5% remaining
private static readonly MIN_REQUESTS_BUFFER = 100; // Keep at least 100 requests as buffer
private static lastNotifiedThreshold: Map<string, number> = new Map(); // Track last notification per user
/**
* Check current rate limit status from GitHub
*/
static async checkGitHubRateLimit(octokit: Octokit, userId: string): Promise<RateLimitInfo> {
try {
const { data } = await octokit.rateLimit.get();
const core = data.rate;
const info: RateLimitInfo = {
limit: core.limit,
remaining: core.remaining,
used: core.used,
reset: new Date(core.reset * 1000),
status: this.calculateStatus(core.remaining, core.limit),
};
// Update database
await this.updateRateLimit(userId, "github", info);
return info;
} catch (error) {
console.error("Failed to check GitHub rate limit:", error);
// Return last known status from database if API check fails
return await this.getLastKnownStatus(userId, "github");
}
}
/**
* Extract rate limit info from response headers
*/
static parseRateLimitHeaders(headers: RateLimitHeaders): Partial<RateLimitInfo> {
const info: Partial<RateLimitInfo> = {};
if (headers["x-ratelimit-limit"]) {
info.limit = parseInt(headers["x-ratelimit-limit"], 10);
}
if (headers["x-ratelimit-remaining"]) {
info.remaining = parseInt(headers["x-ratelimit-remaining"], 10);
}
if (headers["x-ratelimit-used"]) {
info.used = parseInt(headers["x-ratelimit-used"], 10);
}
if (headers["x-ratelimit-reset"]) {
info.reset = new Date(parseInt(headers["x-ratelimit-reset"], 10) * 1000);
}
if (headers["retry-after"]) {
info.retryAfter = parseInt(headers["retry-after"], 10);
}
if (info.remaining !== undefined && info.limit !== undefined) {
info.status = this.calculateStatus(info.remaining, info.limit);
}
return info;
}
/**
* Update rate limit info from API response
*/
static async updateFromResponse(userId: string, headers: RateLimitHeaders): Promise<void> {
const info = this.parseRateLimitHeaders(headers);
if (Object.keys(info).length > 0) {
await this.updateRateLimit(userId, "github", info as RateLimitInfo);
}
}
/**
* Calculate rate limit status based on remaining requests
*/
static calculateStatus(remaining: number, limit: number): RateLimitStatus {
const ratio = remaining / limit;
if (remaining === 0) return "exceeded";
if (remaining < this.MIN_REQUESTS_BUFFER || ratio < this.PAUSE_THRESHOLD) return "limited";
if (ratio < this.WARNING_THRESHOLD) return "warning";
return "ok";
}
/**
* Check if we should pause operations
*/
static async shouldPause(userId: string, provider: "github" | "gitea" = "github"): Promise<boolean> {
const status = await this.getLastKnownStatus(userId, provider);
return status.status === "limited" || status.status === "exceeded";
}
/**
* Calculate wait time until rate limit resets
*/
static calculateWaitTime(reset: Date, retryAfter?: number): number {
if (retryAfter) {
return retryAfter * 1000; // Convert to milliseconds
}
const now = new Date();
const waitTime = reset.getTime() - now.getTime();
return Math.max(0, waitTime);
}
/**
* Wait until rate limit resets
*/
static async waitForReset(userId: string, provider: "github" | "gitea" = "github"): Promise<void> {
const status = await this.getLastKnownStatus(userId, provider);
if (status.status === "ok" || status.status === "warning") {
return; // No need to wait
}
const waitTime = this.calculateWaitTime(status.reset, status.retryAfter);
if (waitTime > 0) {
console.log(`[RateLimit] Waiting ${Math.ceil(waitTime / 1000)}s for rate limit reset...`);
// Create event for UI notification
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "waiting",
provider,
waitTime,
resetAt: status.reset,
message: `API rate limit reached. Waiting ${Math.ceil(waitTime / 1000)} seconds before resuming...`,
},
});
// Wait
await new Promise(resolve => setTimeout(resolve, waitTime));
// Update status after waiting
await this.updateRateLimit(userId, provider, {
...status,
status: "ok",
remaining: status.limit,
used: 0,
});
// Notify that we've resumed
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "resumed",
provider,
message: "Rate limit reset. Resuming operations...",
},
});
}
}
/**
* Update rate limit info in database
*/
private static async updateRateLimit(
userId: string,
provider: "github" | "gitea",
info: RateLimitInfo
): Promise<void> {
const existing = await db
.select()
.from(rateLimits)
.where(and(eq(rateLimits.userId, userId), eq(rateLimits.provider, provider)))
.limit(1);
const data = {
userId,
provider,
limit: info.limit,
remaining: info.remaining,
used: info.used,
reset: info.reset,
retryAfter: info.retryAfter,
status: info.status,
lastChecked: new Date(),
updatedAt: new Date(),
};
if (existing.length > 0) {
await db
.update(rateLimits)
.set(data)
.where(eq(rateLimits.id, existing[0].id));
} else {
await db.insert(rateLimits).values({
id: uuidv4(),
...data,
createdAt: new Date(),
});
}
// Only send notifications at specific thresholds to avoid spam
const usedPercentage = ((info.limit - info.remaining) / info.limit) * 100;
const userKey = `${userId}-${provider}`;
const lastNotified = this.lastNotifiedThreshold.get(userKey) || 0;
// Notify at 80% usage (20% remaining)
if (usedPercentage >= 80 && usedPercentage < 100 && lastNotified < 80) {
this.lastNotifiedThreshold.set(userKey, 80);
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "warning",
provider,
status: info.status,
remaining: info.remaining,
limit: info.limit,
usedPercentage: Math.round(usedPercentage),
message: `GitHub API rate limit at ${Math.round(usedPercentage)}%. ${info.remaining} requests remaining.`,
},
});
console.log(`[RateLimit] 80% threshold reached for user ${userId}: ${info.remaining}/${info.limit} requests remaining`);
}
// Notify at 100% usage (0 remaining)
if (info.remaining === 0 && lastNotified < 100) {
this.lastNotifiedThreshold.set(userKey, 100);
const resetTime = new Date(info.reset);
const minutesUntilReset = Math.ceil((resetTime.getTime() - Date.now()) / 60000);
await publishEvent({
userId,
channel: "rate-limit",
payload: {
type: "exceeded",
provider,
status: "exceeded",
remaining: 0,
limit: info.limit,
usedPercentage: 100,
reset: info.reset,
message: `GitHub API rate limit exceeded. Will automatically resume in ${minutesUntilReset} minutes.`,
},
});
console.log(`[RateLimit] 100% rate limit exceeded for user ${userId}. Resets at ${resetTime.toLocaleTimeString()}`);
}
// Reset notification threshold when rate limit resets
if (info.remaining > info.limit * 0.5 && lastNotified > 0) {
this.lastNotifiedThreshold.delete(userKey);
}
}
/**
* Get last known rate limit status from database
*/
private static async getLastKnownStatus(
userId: string,
provider: "github" | "gitea"
): Promise<RateLimitInfo> {
const [result] = await db
.select()
.from(rateLimits)
.where(and(eq(rateLimits.userId, userId), eq(rateLimits.provider, provider)))
.limit(1);
if (result) {
return {
limit: result.limit,
remaining: result.remaining,
used: result.used,
reset: result.reset,
retryAfter: result.retryAfter ?? undefined,
status: result.status as RateLimitStatus,
};
}
// Return default if no data
return {
limit: 5000,
remaining: 5000,
used: 0,
reset: new Date(Date.now() + 3600000), // 1 hour from now
status: "ok",
};
}
/**
* Get human-readable status message
*/
private static getStatusMessage(info: RateLimitInfo): string {
const percentage = Math.round((info.remaining / info.limit) * 100);
switch (info.status) {
case "exceeded":
return `API rate limit exceeded. Resets at ${info.reset.toLocaleTimeString()}.`;
case "limited":
return `API rate limit critical: Only ${info.remaining} requests remaining (${percentage}%). Pausing operations...`;
case "warning":
return `API rate limit warning: ${info.remaining} requests remaining (${percentage}%).`;
default:
return `API rate limit healthy: ${info.remaining}/${info.limit} requests remaining.`;
}
}
/**
* Smart retry with exponential backoff for rate-limited requests
*/
static async retryWithBackoff<T>(
fn: () => Promise<T>,
userId: string,
maxRetries: number = 3
): Promise<T> {
let lastError: any;
for (let attempt = 0; attempt < maxRetries; attempt++) {
try {
// Check if we should pause before attempting
if (await this.shouldPause(userId)) {
await this.waitForReset(userId);
}
return await fn();
} catch (error: any) {
lastError = error;
// Check if it's a rate limit error
if (error.status === 403 && error.message?.includes("rate limit")) {
console.log(`[RateLimit] Rate limit hit on attempt ${attempt + 1}/${maxRetries}`);
// Parse rate limit headers from error response if available
if (error.response?.headers) {
await this.updateFromResponse(userId, error.response.headers);
}
// Wait for reset
await this.waitForReset(userId);
} else if (error.status === 429) {
// Too Many Requests - use exponential backoff
const backoffTime = Math.min(1000 * Math.pow(2, attempt), 30000); // Max 30s
console.log(`[RateLimit] Too many requests, backing off ${backoffTime}ms`);
await new Promise(resolve => setTimeout(resolve, backoffTime));
} else {
// Not a rate limit error, throw immediately
throw error;
}
}
}
throw lastError;
}
}
/**
* Middleware to check rate limits before making API calls
*/
export async function withRateLimitCheck<T>(
userId: string,
operation: () => Promise<T>,
operationName: string = "API call"
): Promise<T> {
// Check if we should pause
if (await RateLimitManager.shouldPause(userId)) {
console.log(`[RateLimit] Pausing ${operationName} due to rate limit`);
await RateLimitManager.waitForReset(userId);
}
// Execute with retry logic
return await RateLimitManager.retryWithBackoff(operation, userId);
}
/**
* Hook to update rate limits from Octokit responses
*/
export function createOctokitRateLimitPlugin(userId: string) {
return {
hook: (request: any, options: any) => {
return request(options).then((response: any) => {
// Update rate limit from response headers
if (response.headers) {
RateLimitManager.updateFromResponse(userId, response.headers).catch(console.error);
}
return response;
});
},
};
}

View File

@@ -260,11 +260,13 @@ async function recoverMirrorJob(job: any, remainingItemIds: string[]) {
throw new Error('GitHub token not found in configuration');
}
// Create GitHub client with error handling
// Create GitHub client with error handling and rate limit tracking
let octokit;
try {
const decryptedToken = getDecryptedGitHubToken(config);
octokit = createGitHubClient(decryptedToken);
const githubUsername = config.githubConfig?.owner || undefined;
const userId = config.userId || undefined;
octokit = createGitHubClient(decryptedToken, userId, githubUsername);
} catch (error) {
throw new Error(`Failed to create GitHub client: ${error instanceof Error ? error.message : String(error)}`);
}

View File

@@ -0,0 +1,75 @@
import { describe, it, expect } from 'bun:test';
import { mergeGitReposPreferStarred, normalizeGitRepoToInsert, calcBatchSizeForInsert } from '@/lib/repo-utils';
import type { GitRepo } from '@/types/Repository';
function sampleRepo(overrides: Partial<GitRepo> = {}): GitRepo {
const base: GitRepo = {
name: 'repo',
fullName: 'owner/repo',
url: 'https://github.com/owner/repo',
cloneUrl: 'https://github.com/owner/repo.git',
owner: 'owner',
organization: undefined,
mirroredLocation: '',
destinationOrg: null,
isPrivate: false,
isForked: false,
forkedFrom: undefined,
hasIssues: true,
isStarred: false,
isArchived: false,
size: 1,
hasLFS: false,
hasSubmodules: false,
language: null,
description: null,
defaultBranch: 'main',
visibility: 'public',
status: 'imported',
lastMirrored: undefined,
errorMessage: undefined,
createdAt: new Date(),
updatedAt: new Date(),
};
return { ...base, ...overrides };
}
describe('mergeGitReposPreferStarred', () => {
it('keeps unique repos', () => {
const basic = [sampleRepo({ fullName: 'a/x', name: 'x' })];
const starred: GitRepo[] = [];
const merged = mergeGitReposPreferStarred(basic, starred);
expect(merged).toHaveLength(1);
expect(merged[0].fullName).toBe('a/x');
});
it('prefers starred when duplicate exists', () => {
const basic = [sampleRepo({ fullName: 'a/x', name: 'x', isStarred: false })];
const starred = [sampleRepo({ fullName: 'a/x', name: 'x', isStarred: true })];
const merged = mergeGitReposPreferStarred(basic, starred);
expect(merged).toHaveLength(1);
expect(merged[0].isStarred).toBe(true);
});
});
describe('normalizeGitRepoToInsert', () => {
it('sets undefined optional fields to null', () => {
const repo = sampleRepo({ organization: undefined, forkedFrom: undefined, language: undefined, description: undefined, lastMirrored: undefined, errorMessage: undefined });
const insert = normalizeGitRepoToInsert(repo, { userId: 'u', configId: 'c' });
expect(insert.organization).toBeNull();
expect(insert.forkedFrom).toBeNull();
expect(insert.language).toBeNull();
expect(insert.description).toBeNull();
expect(insert.lastMirrored).toBeNull();
expect(insert.errorMessage).toBeNull();
});
});
describe('calcBatchSizeForInsert', () => {
it('respects 999 parameter limit', () => {
const batch = calcBatchSizeForInsert(29);
expect(batch).toBeGreaterThan(0);
expect(batch * 29).toBeLessThanOrEqual(999);
});
});

71
src/lib/repo-utils.ts Normal file
View File

@@ -0,0 +1,71 @@
import { v4 as uuidv4 } from 'uuid';
import type { GitRepo } from '@/types/Repository';
import { repositories } from '@/lib/db/schema';
export type RepoInsert = typeof repositories.$inferInsert;
// Merge lists and de-duplicate by fullName, preferring starred variant when present
export function mergeGitReposPreferStarred(
basicAndForked: GitRepo[],
starred: GitRepo[]
): GitRepo[] {
const map = new Map<string, GitRepo>();
for (const r of [...basicAndForked, ...starred]) {
const existing = map.get(r.fullName);
if (!existing || (!existing.isStarred && r.isStarred)) {
map.set(r.fullName, r);
}
}
return Array.from(map.values());
}
// Convert a GitRepo to a normalized DB insert object with all nullable fields set
export function normalizeGitRepoToInsert(
repo: GitRepo,
{
userId,
configId,
}: { userId: string; configId: string }
): RepoInsert {
return {
id: uuidv4(),
userId,
configId,
name: repo.name,
fullName: repo.fullName,
url: repo.url,
cloneUrl: repo.cloneUrl,
owner: repo.owner,
organization: repo.organization ?? null,
mirroredLocation: repo.mirroredLocation || '',
destinationOrg: repo.destinationOrg || null,
isPrivate: repo.isPrivate,
isForked: repo.isForked,
forkedFrom: repo.forkedFrom ?? null,
hasIssues: repo.hasIssues,
isStarred: repo.isStarred,
isArchived: repo.isArchived,
size: repo.size,
hasLFS: repo.hasLFS,
hasSubmodules: repo.hasSubmodules,
language: repo.language ?? null,
description: repo.description ?? null,
defaultBranch: repo.defaultBranch,
visibility: repo.visibility,
status: 'imported',
lastMirrored: repo.lastMirrored ?? null,
errorMessage: repo.errorMessage ?? null,
createdAt: repo.createdAt || new Date(),
updatedAt: repo.updatedAt || new Date(),
};
}
// Compute a safe batch size based on SQLite 999-parameter limit
export function calcBatchSizeForInsert(columnCount: number, maxParams = 999): number {
if (columnCount <= 0) return 1;
// Reserve a little headroom in case column count drifts
const safety = 0;
const effectiveMax = Math.max(1, maxParams - safety);
return Math.max(1, Math.floor(effectiveMax / columnCount));
}

View File

@@ -7,7 +7,7 @@
import { db, configs, repositories } from '@/lib/db';
import { eq, and, or, sql, not, inArray } from 'drizzle-orm';
import { createGitHubClient, getGithubRepositories, getGithubStarredRepositories } from '@/lib/github';
import { createGiteaClient, deleteGiteaRepo, archiveGiteaRepo } from '@/lib/gitea';
import { createGiteaClient, deleteGiteaRepo, archiveGiteaRepo, getGiteaRepoOwnerAsync, checkRepoLocation } from '@/lib/gitea';
import { getDecryptedGitHubToken, getDecryptedGiteaToken } from '@/lib/utils/config-encryption';
import { publishEvent } from '@/lib/events';
@@ -23,19 +23,42 @@ async function identifyOrphanedRepositories(config: any): Promise<any[]> {
const userId = config.userId;
try {
// Get current GitHub repositories
// Get current GitHub repositories with rate limit tracking
const decryptedToken = getDecryptedGitHubToken(config);
const octokit = createGitHubClient(decryptedToken);
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedToken, userId, githubUsername);
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
config.githubConfig?.includeStarred
? getGithubStarredRepositories({ octokit, config })
: Promise.resolve([]),
]);
let allGithubRepos = [];
let githubApiAccessible = true;
try {
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
config.githubConfig?.includeStarred
? getGithubStarredRepositories({ octokit, config })
: Promise.resolve([]),
]);
allGithubRepos = [...basicAndForkedRepos, ...starredRepos];
} catch (githubError: any) {
// Handle GitHub API errors gracefully
console.warn(`[Repository Cleanup] GitHub API error for user ${userId}: ${githubError.message}`);
// Check if it's a critical error (like account deleted/banned)
if (githubError.status === 404 || githubError.status === 403) {
console.error(`[Repository Cleanup] CRITICAL: GitHub account may be deleted/banned. Skipping cleanup to prevent data loss.`);
console.error(`[Repository Cleanup] Consider using CLEANUP_ORPHANED_REPO_ACTION=archive instead of delete for safety.`);
// Return empty array to skip cleanup entirely when GitHub account is inaccessible
return [];
}
// For other errors, also skip cleanup to be safe
console.error(`[Repository Cleanup] Skipping cleanup due to GitHub API error. This prevents accidental deletion of backups.`);
return [];
}
const allGithubRepos = [...basicAndForkedRepos, ...starredRepos];
const githubRepoFullNames = new Set(allGithubRepos.map(repo => repo.fullName));
// Get all repositories from our database
@@ -44,13 +67,32 @@ async function identifyOrphanedRepositories(config: any): Promise<any[]> {
.from(repositories)
.where(eq(repositories.userId, userId));
// Identify orphaned repositories
const orphanedRepos = dbRepos.filter(repo => !githubRepoFullNames.has(repo.fullName));
// Only identify repositories as orphaned if we successfully accessed GitHub
// This prevents false positives when GitHub is down or account is inaccessible
const orphanedRepos = dbRepos.filter(repo => {
const isOrphaned = !githubRepoFullNames.has(repo.fullName);
if (!isOrphaned) {
return false;
}
// Skip repositories we've already archived/preserved
if (repo.status === 'archived' || repo.isArchived) {
console.log(`[Repository Cleanup] Skipping ${repo.fullName} - already archived`);
return false;
}
return true;
});
if (orphanedRepos.length > 0) {
console.log(`[Repository Cleanup] Found ${orphanedRepos.length} orphaned repositories for user ${userId}`);
}
return orphanedRepos;
} catch (error) {
console.error(`[Repository Cleanup] Error identifying orphaned repositories for user ${userId}:`, error);
throw error;
// Return empty array on error to prevent accidental deletions
return [];
}
}
@@ -69,7 +111,12 @@ async function handleOrphanedRepository(
console.log(`[Repository Cleanup] Skipping orphaned repository ${repoFullName}`);
return;
}
if (repo.status === 'archived' || repo.isArchived) {
console.log(`[Repository Cleanup] Repository ${repoFullName} already archived; skipping additional actions`);
return;
}
if (dryRun) {
console.log(`[Repository Cleanup] DRY RUN: Would ${action} orphaned repository ${repoFullName}`);
return;
@@ -80,26 +127,46 @@ async function handleOrphanedRepository(
const giteaToken = getDecryptedGiteaToken(config);
const giteaClient = createGiteaClient(config.giteaConfig.url, giteaToken);
// Determine the Gitea owner and repo name
const mirroredLocation = repo.mirroredLocation || '';
let giteaOwner = repo.owner;
let giteaRepoName = repo.name;
if (mirroredLocation) {
const parts = mirroredLocation.split('/');
if (parts.length >= 2) {
giteaOwner = parts[parts.length - 2];
giteaRepoName = parts[parts.length - 1];
}
// Determine the Gitea owner and repo name more robustly
const mirroredLocation = (repo.mirroredLocation || '').trim();
let giteaOwner: string;
let giteaRepoName: string;
if (mirroredLocation && mirroredLocation.includes('/')) {
const [ownerPart, namePart] = mirroredLocation.split('/');
giteaOwner = ownerPart;
giteaRepoName = namePart;
} else {
// Fall back to expected owner based on config and repo flags (starred/org overrides)
giteaOwner = await getGiteaRepoOwnerAsync({ config, repository: repo });
giteaRepoName = repo.name;
}
// Normalize owner casing to avoid GetUserByName issues on some Gitea setups
giteaOwner = giteaOwner.trim();
if (action === 'archive') {
console.log(`[Repository Cleanup] Archiving orphaned repository ${repoFullName} in Gitea`);
// Best-effort check to validate actual location; falls back gracefully
try {
const { present, actualOwner } = await checkRepoLocation({
config,
repository: repo,
expectedOwner: giteaOwner,
});
if (present) {
giteaOwner = actualOwner;
}
} catch {
// Non-fatal; continue with best guess
}
await archiveGiteaRepo(giteaClient, giteaOwner, giteaRepoName);
// Update database status
await db.update(repositories).set({
status: 'archived',
isArchived: true,
errorMessage: 'Repository archived - no longer in GitHub',
updatedAt: new Date(),
}).where(eq(repositories.id, repo.id));
@@ -211,7 +278,7 @@ async function runRepositoryCleanup(config: any): Promise<{
// Process orphaned repositories
const action = cleanupConfig.orphanedRepoAction || 'archive';
const dryRun = cleanupConfig.dryRun ?? true;
const dryRun = cleanupConfig.dryRun ?? false;
const batchSize = cleanupConfig.batchSize || 10;
const pauseBetweenDeletes = cleanupConfig.pauseBetweenDeletes || 2000;
@@ -348,6 +415,9 @@ export function isRepositoryCleanupServiceRunning(): boolean {
return cleanupInterval !== null;
}
// Export functions for use by scheduler
export { identifyOrphanedRepositories, handleOrphanedRepository };
/**
* Manually trigger repository cleanup for a specific user
*/
@@ -370,4 +440,4 @@ export async function triggerRepositoryCleanup(userId: string): Promise<{
}
return runRepositoryCleanup(config);
}
}

View File

@@ -5,16 +5,17 @@
*/
import { db, configs, repositories } from '@/lib/db';
import { eq, and, or, lt, gte } from 'drizzle-orm';
import { syncGiteaRepo } from '@/lib/gitea';
import { createGitHubClient } from '@/lib/github';
import { eq, and, or } from 'drizzle-orm';
import { syncGiteaRepo, mirrorGithubRepoToGitea } from '@/lib/gitea';
import { getDecryptedGitHubToken } from '@/lib/utils/config-encryption';
import { parseInterval, formatDuration } from '@/lib/utils/duration-parser';
import type { Repository } from '@/lib/db/schema';
import { repoStatusEnum, repositoryVisibilityEnum } from '@/types/Repository';
import { mergeGitReposPreferStarred, normalizeGitRepoToInsert, calcBatchSizeForInsert } from '@/lib/repo-utils';
let schedulerInterval: NodeJS.Timeout | null = null;
let isSchedulerRunning = false;
let hasPerformedAutoStart = false; // Track if we've already done auto-start
/**
* Parse schedule interval with enhanced support for duration strings, cron, and numbers
@@ -41,6 +42,12 @@ async function runScheduledSync(config: any): Promise<void> {
console.log(`[Scheduler] Running scheduled sync for user ${userId}`);
try {
// Check if tokens are configured before proceeding
if (!config.githubConfig?.token || !config.giteaConfig?.token) {
console.log(`[Scheduler] Skipping sync for user ${userId}: GitHub or Gitea tokens not configured`);
return;
}
// Update lastRun timestamp
const currentTime = new Date();
const scheduleConfig = config.scheduleConfig || {};
@@ -68,6 +75,166 @@ async function runScheduledSync(config: any): Promise<void> {
updatedAt: currentTime,
}).where(eq(configs.id, config.id));
// Auto-discovery: Check for new GitHub repositories
if (scheduleConfig.autoImport !== false) {
console.log(`[Scheduler] Checking for new GitHub repositories for user ${userId}...`);
try {
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
const { v4: uuidv4 } = await import('uuid');
const { getDecryptedGitHubToken } = await import('@/lib/utils/config-encryption');
// Create GitHub client
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
config.githubConfig?.includeStarred
? getGithubStarredRepositories({ octokit, config })
: Promise.resolve([]),
]);
const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos);
// Check for new repositories
const existingRepos = await db
.select({ fullName: repositories.fullName })
.from(repositories)
.where(eq(repositories.userId, userId));
const existingRepoNames = new Set(existingRepos.map(r => r.fullName));
const newRepos = allGithubRepos.filter(r => !existingRepoNames.has(r.fullName));
if (newRepos.length > 0) {
console.log(`[Scheduler] Found ${newRepos.length} new repositories for user ${userId}`);
// Insert new repositories
const reposToInsert = newRepos.map(repo =>
normalizeGitRepoToInsert(repo, { userId, configId: config.id })
);
// Batch insert to avoid SQLite parameter limit
const sample = reposToInsert[0];
const columnCount = Object.keys(sample ?? {}).length || 1;
const BATCH_SIZE = calcBatchSizeForInsert(columnCount);
for (let i = 0; i < reposToInsert.length; i += BATCH_SIZE) {
const batch = reposToInsert.slice(i, i + BATCH_SIZE);
await db
.insert(repositories)
.values(batch)
.onConflictDoNothing({ target: [repositories.userId, repositories.fullName] });
}
console.log(`[Scheduler] Successfully imported ${newRepos.length} new repositories for user ${userId}`);
} else {
console.log(`[Scheduler] No new repositories found for user ${userId}`);
}
} catch (error) {
console.error(`[Scheduler] Failed to auto-import repositories for user ${userId}:`, error);
}
}
// Auto-cleanup: Remove orphaned repositories (repos that no longer exist in GitHub)
if (config.cleanupConfig?.deleteIfNotInGitHub) {
console.log(`[Scheduler] Checking for orphaned repositories to cleanup for user ${userId}...`);
try {
const { identifyOrphanedRepositories, handleOrphanedRepository } = await import('@/lib/repository-cleanup-service');
const orphanedRepos = await identifyOrphanedRepositories(config);
if (orphanedRepos.length > 0) {
console.log(`[Scheduler] Found ${orphanedRepos.length} orphaned repositories for cleanup`);
for (const repo of orphanedRepos) {
try {
await handleOrphanedRepository(
config,
repo,
config.cleanupConfig.orphanedRepoAction || 'archive',
config.cleanupConfig.dryRun ?? false
);
console.log(`[Scheduler] Handled orphaned repository: ${repo.fullName}`);
} catch (error) {
console.error(`[Scheduler] Failed to handle orphaned repository ${repo.fullName}:`, error);
}
}
} else {
console.log(`[Scheduler] No orphaned repositories found for cleanup`);
}
} catch (error) {
console.error(`[Scheduler] Failed to cleanup orphaned repositories for user ${userId}:`, error);
}
}
// Auto-mirror: Mirror imported/pending/failed repositories if enabled
if (scheduleConfig.autoMirror) {
try {
console.log(`[Scheduler] Auto-mirror enabled - checking for repositories to mirror for user ${userId}...`);
const reposNeedingMirror = await db
.select()
.from(repositories)
.where(
and(
eq(repositories.userId, userId),
or(
eq(repositories.status, 'imported'),
eq(repositories.status, 'pending'),
eq(repositories.status, 'failed')
)
)
);
if (reposNeedingMirror.length > 0) {
console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need initial mirroring`);
// Prepare Octokit client
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
// Process repositories in batches
const batchSize = scheduleConfig.batchSize || 10;
const pauseBetweenBatches = scheduleConfig.pauseBetweenBatches || 2000;
for (let i = 0; i < reposNeedingMirror.length; i += batchSize) {
const batch = reposNeedingMirror.slice(i, Math.min(i + batchSize, reposNeedingMirror.length));
console.log(`[Scheduler] Auto-mirror batch ${Math.floor(i / batchSize) + 1} of ${Math.ceil(reposNeedingMirror.length / batchSize)} (${batch.length} repos)`);
await Promise.all(
batch.map(async (repo) => {
try {
const repository: Repository = {
...repo,
status: repoStatusEnum.parse(repo.status),
organization: repo.organization ?? undefined,
lastMirrored: repo.lastMirrored ?? undefined,
errorMessage: repo.errorMessage ?? undefined,
mirroredLocation: repo.mirroredLocation || '',
forkedFrom: repo.forkedFrom ?? undefined,
visibility: repositoryVisibilityEnum.parse(repo.visibility),
};
await mirrorGithubRepoToGitea({ octokit, repository, config });
console.log(`[Scheduler] Auto-mirrored repository: ${repo.fullName}`);
} catch (error) {
console.error(`[Scheduler] Failed to auto-mirror repository ${repo.fullName}:`, error);
}
})
);
// Pause between batches if configured
if (i + batchSize < reposNeedingMirror.length) {
console.log(`[Scheduler] Pausing for ${pauseBetweenBatches}ms before next auto-mirror batch...`);
await new Promise(resolve => setTimeout(resolve, pauseBetweenBatches));
}
}
} else {
console.log(`[Scheduler] No repositories need initial mirroring`);
}
} catch (mirrorError) {
console.error(`[Scheduler] Error during auto-mirror phase for user ${userId}:`, mirrorError);
}
}
// Get repositories to sync
let reposToSync = await db
.select()
@@ -176,6 +343,265 @@ async function syncSingleRepository(config: any, repo: any): Promise<void> {
}
}
/**
* Check if we should auto-start based on environment configuration
*/
async function checkAutoStartConfiguration(): Promise<boolean> {
// Don't auto-start more than once
if (hasPerformedAutoStart) {
return false;
}
try {
// Check if any configuration has scheduling enabled or mirror interval set
const activeConfigs = await db
.select()
.from(configs)
.where(eq(configs.isActive, true));
for (const config of activeConfigs) {
// Check if scheduling is enabled via environment
const scheduleEnabled = config.scheduleConfig?.enabled === true;
const hasMirrorInterval = !!config.giteaConfig?.mirrorInterval;
// If either SCHEDULE_ENABLED=true or GITEA_MIRROR_INTERVAL is set, we should auto-start
if (scheduleEnabled || hasMirrorInterval) {
console.log(`[Scheduler] Auto-start conditions met for user ${config.userId} (scheduleEnabled=${scheduleEnabled}, hasMirrorInterval=${hasMirrorInterval})`);
return true;
}
}
return false;
} catch (error) {
console.error('[Scheduler] Error checking auto-start configuration:', error);
return false;
}
}
/**
* Perform initial auto-start: import repositories and trigger mirror
*/
async function performInitialAutoStart(): Promise<void> {
hasPerformedAutoStart = true;
try {
console.log('[Scheduler] Performing initial auto-start...');
// Get all active configurations
const activeConfigs = await db
.select()
.from(configs)
.where(eq(configs.isActive, true));
for (const config of activeConfigs) {
// Skip if tokens are not configured
if (!config.githubConfig?.token || !config.giteaConfig?.token) {
console.log(`[Scheduler] Skipping auto-start for user ${config.userId}: tokens not configured`);
continue;
}
const scheduleEnabled = config.scheduleConfig?.enabled === true;
const hasMirrorInterval = !!config.giteaConfig?.mirrorInterval;
// Only process configs that have scheduling or mirror interval configured
if (!scheduleEnabled && !hasMirrorInterval) {
continue;
}
console.log(`[Scheduler] Auto-starting for user ${config.userId}...`);
try {
// Step 1: Import repositories from GitHub
console.log(`[Scheduler] Step 1: Importing repositories from GitHub for user ${config.userId}...`);
const { getGithubRepositories, getGithubStarredRepositories } = await import('@/lib/github');
const { v4: uuidv4 } = await import('uuid');
// Create GitHub client
const decryptedToken = getDecryptedGitHubToken(config);
const { Octokit } = await import('@octokit/rest');
const octokit = new Octokit({ auth: decryptedToken });
// Fetch GitHub data
const [basicAndForkedRepos, starredRepos] = await Promise.all([
getGithubRepositories({ octokit, config }),
config.githubConfig?.includeStarred
? getGithubStarredRepositories({ octokit, config })
: Promise.resolve([]),
]);
const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos);
// Check for new repositories
const existingRepos = await db
.select({ fullName: repositories.fullName })
.from(repositories)
.where(eq(repositories.userId, config.userId));
const existingRepoNames = new Set(existingRepos.map(r => r.fullName));
const reposToImport = allGithubRepos.filter(r => !existingRepoNames.has(r.fullName));
if (reposToImport.length > 0) {
console.log(`[Scheduler] Importing ${reposToImport.length} repositories for user ${config.userId}...`);
// Insert new repositories
const reposToInsert = reposToImport.map(repo =>
normalizeGitRepoToInsert(repo, { userId: config.userId, configId: config.id })
);
// Batch insert to avoid SQLite parameter limit
const sample = reposToInsert[0];
const columnCount = Object.keys(sample ?? {}).length || 1;
const BATCH_SIZE = calcBatchSizeForInsert(columnCount);
for (let i = 0; i < reposToInsert.length; i += BATCH_SIZE) {
const batch = reposToInsert.slice(i, i + BATCH_SIZE);
await db
.insert(repositories)
.values(batch)
.onConflictDoNothing({ target: [repositories.userId, repositories.fullName] });
}
console.log(`[Scheduler] Successfully imported ${reposToImport.length} repositories`);
} else {
console.log(`[Scheduler] No new repositories to import for user ${config.userId}`);
}
// Check if we already have mirrored repositories (indicating this isn't first run)
const mirroredRepos = await db
.select()
.from(repositories)
.where(
and(
eq(repositories.userId, config.userId),
or(
eq(repositories.status, 'mirrored'),
eq(repositories.status, 'synced')
)
)
)
.limit(1);
// If we already have mirrored repos, skip the initial mirror (let regular sync handle it)
if (mirroredRepos.length > 0) {
console.log(`[Scheduler] User ${config.userId} already has mirrored repositories, skipping initial mirror (let regular sync handle updates)`);
// Still update the schedule config to indicate scheduling is active
const currentTime = new Date();
const intervalSource = config.scheduleConfig?.interval ||
config.giteaConfig?.mirrorInterval ||
'8h';
const interval = parseScheduleInterval(intervalSource);
const nextRun = new Date(currentTime.getTime() + interval);
await db.update(configs).set({
scheduleConfig: {
...config.scheduleConfig,
enabled: true,
lastRun: currentTime,
nextRun: nextRun,
},
updatedAt: currentTime,
}).where(eq(configs.id, config.id));
console.log(`[Scheduler] Scheduling enabled for user ${config.userId}, next sync at ${nextRun.toISOString()}`);
continue;
}
// Step 2: Trigger mirror for all repositories that need mirroring
console.log(`[Scheduler] Step 2: Triggering mirror for repositories that need mirroring...`);
const reposNeedingMirror = await db
.select()
.from(repositories)
.where(
and(
eq(repositories.userId, config.userId),
or(
eq(repositories.status, 'imported'),
eq(repositories.status, 'pending'),
eq(repositories.status, 'failed')
)
)
);
if (reposNeedingMirror.length > 0) {
console.log(`[Scheduler] Found ${reposNeedingMirror.length} repositories that need mirroring`);
// Reuse the octokit instance from above
// (octokit was already created in the import phase)
// Process repositories in batches
const batchSize = config.scheduleConfig?.batchSize || 5;
for (let i = 0; i < reposNeedingMirror.length; i += batchSize) {
const batch = reposNeedingMirror.slice(i, Math.min(i + batchSize, reposNeedingMirror.length));
console.log(`[Scheduler] Processing batch ${Math.floor(i / batchSize) + 1} of ${Math.ceil(reposNeedingMirror.length / batchSize)} (${batch.length} repos)`);
await Promise.all(
batch.map(async (repo) => {
try {
const repository: Repository = {
...repo,
status: repoStatusEnum.parse(repo.status),
organization: repo.organization ?? undefined,
lastMirrored: repo.lastMirrored ?? undefined,
errorMessage: repo.errorMessage ?? undefined,
mirroredLocation: repo.mirroredLocation || '',
forkedFrom: repo.forkedFrom ?? undefined,
visibility: repositoryVisibilityEnum.parse(repo.visibility),
};
await mirrorGithubRepoToGitea({
octokit,
repository,
config
});
console.log(`[Scheduler] Successfully mirrored repository: ${repo.fullName}`);
} catch (error) {
console.error(`[Scheduler] Failed to mirror repository ${repo.fullName}:`, error);
}
})
);
// Pause between batches if configured
if (i + batchSize < reposNeedingMirror.length) {
const pauseTime = config.scheduleConfig?.pauseBetweenBatches || 2000;
console.log(`[Scheduler] Pausing for ${pauseTime}ms before next batch...`);
await new Promise(resolve => setTimeout(resolve, pauseTime));
}
}
console.log(`[Scheduler] Completed initial mirror for ${reposNeedingMirror.length} repositories`);
} else {
console.log(`[Scheduler] No repositories need mirroring`);
}
// Update the schedule config to indicate we've run
const currentTime = new Date();
const intervalSource = config.scheduleConfig?.interval ||
config.giteaConfig?.mirrorInterval ||
'8h';
const interval = parseScheduleInterval(intervalSource);
const nextRun = new Date(currentTime.getTime() + interval);
await db.update(configs).set({
scheduleConfig: {
...config.scheduleConfig,
enabled: true, // Ensure scheduling is enabled
lastRun: currentTime,
nextRun: nextRun,
},
updatedAt: currentTime,
}).where(eq(configs.id, config.id));
console.log(`[Scheduler] Auto-start completed for user ${config.userId}, next sync at ${nextRun.toISOString()}`);
} catch (error) {
console.error(`[Scheduler] Failed to auto-start for user ${config.userId}:`, error);
}
}
console.log('[Scheduler] Initial auto-start completed');
} catch (error) {
console.error('[Scheduler] Failed to perform initial auto-start:', error);
}
}
/**
* Main scheduler loop
*/
@@ -202,25 +628,41 @@ async function schedulerLoop(): Promise<void> {
config.scheduleConfig?.enabled === true
);
if (enabledConfigs.length === 0) {
console.log(`[Scheduler] No configurations with scheduling enabled (found ${activeConfigs.length} active configs)`);
// Further filter configs that have valid tokens
const validConfigs = enabledConfigs.filter(config => {
const hasGitHubToken = !!config.githubConfig?.token;
const hasGiteaToken = !!config.giteaConfig?.token;
// Show details about why configs are not enabled
activeConfigs.forEach(config => {
const scheduleEnabled = config.scheduleConfig?.enabled;
const mirrorInterval = config.giteaConfig?.mirrorInterval;
console.log(`[Scheduler] User ${config.userId}: scheduleEnabled=${scheduleEnabled}, mirrorInterval=${mirrorInterval}`);
});
if (!hasGitHubToken || !hasGiteaToken) {
console.log(`[Scheduler] User ${config.userId}: Scheduling enabled but tokens missing (GitHub: ${hasGitHubToken}, Gitea: ${hasGiteaToken})`);
return false;
}
return true;
});
if (validConfigs.length === 0) {
if (enabledConfigs.length > 0) {
console.log(`[Scheduler] ${enabledConfigs.length} config(s) have scheduling enabled but lack required tokens`);
} else {
console.log(`[Scheduler] No configurations with scheduling enabled (found ${activeConfigs.length} active configs)`);
// Show details about why configs are not enabled
activeConfigs.forEach(config => {
const scheduleEnabled = config.scheduleConfig?.enabled;
const mirrorInterval = config.giteaConfig?.mirrorInterval;
console.log(`[Scheduler] User ${config.userId}: scheduleEnabled=${scheduleEnabled}, mirrorInterval=${mirrorInterval}`);
});
}
return;
}
console.log(`[Scheduler] Processing ${enabledConfigs.length} configurations with scheduling enabled (out of ${activeConfigs.length} total active configs)`);
console.log(`[Scheduler] Processing ${validConfigs.length} valid configurations (out of ${enabledConfigs.length} with scheduling enabled)`);
// Check each configuration to see if it's time to run
const currentTime = new Date();
for (const config of enabledConfigs) {
for (const config of validConfigs) {
const scheduleConfig = config.scheduleConfig || {};
// Check if it's time to run based on nextRun
@@ -242,7 +684,7 @@ async function schedulerLoop(): Promise<void> {
/**
* Start the scheduler service
*/
export function startSchedulerService(): void {
export async function startSchedulerService(): Promise<void> {
if (schedulerInterval) {
console.log('[Scheduler] Scheduler service is already running');
return;
@@ -250,6 +692,14 @@ export function startSchedulerService(): void {
console.log('[Scheduler] Starting scheduler service');
// Check if we should auto-start mirroring based on environment variables
const shouldAutoStart = await checkAutoStartConfiguration();
if (shouldAutoStart) {
console.log('[Scheduler] Auto-start detected from environment variables, triggering initial import and mirror...');
await performInitialAutoStart();
}
// Run immediately on start
schedulerLoop().catch(error => {
console.error('[Scheduler] Error during initial scheduler run:', error);
@@ -283,4 +733,4 @@ export function stopSchedulerService(): void {
*/
export function isSchedulerServiceRunning(): boolean {
return schedulerInterval !== null;
}
}

View File

@@ -0,0 +1,56 @@
import { describe, expect, it } from "bun:test";
import { normalizeOidcProviderConfig, OidcConfigError } from "./oidc-config";
const issuer = "https://auth.example.com";
describe("normalizeOidcProviderConfig", () => {
it("returns provided endpoints when complete", async () => {
const result = await normalizeOidcProviderConfig(issuer, {
clientId: "client",
clientSecret: "secret",
authorizationEndpoint: "https://auth.example.com/auth",
tokenEndpoint: "https://auth.example.com/token",
jwksEndpoint: "https://auth.example.com/jwks",
userInfoEndpoint: "https://auth.example.com/userinfo",
scopes: ["openid", "email"],
pkce: false,
}, async () => {
throw new Error("fetch should not be called when endpoints are provided");
});
expect(result.oidcConfig.authorizationEndpoint).toBe("https://auth.example.com/auth");
expect(result.oidcConfig.tokenEndpoint).toBe("https://auth.example.com/token");
expect(result.oidcConfig.jwksEndpoint).toBe("https://auth.example.com/jwks");
expect(result.oidcConfig.userInfoEndpoint).toBe("https://auth.example.com/userinfo");
expect(result.oidcConfig.scopes).toEqual(["openid", "email"]);
expect(result.oidcConfig.pkce).toBe(false);
});
it("derives missing fields from discovery", async () => {
const fetchMock = async () =>
new Response(JSON.stringify({
authorization_endpoint: "https://auth.example.com/auth",
token_endpoint: "https://auth.example.com/token",
jwks_uri: "https://auth.example.com/jwks",
userinfo_endpoint: "https://auth.example.com/userinfo",
scopes_supported: ["openid", "email", "profile"],
}));
const result = await normalizeOidcProviderConfig(issuer, {
clientId: "client",
clientSecret: "secret",
}, fetchMock);
expect(result.oidcConfig.authorizationEndpoint).toBe("https://auth.example.com/auth");
expect(result.oidcConfig.tokenEndpoint).toBe("https://auth.example.com/token");
expect(result.oidcConfig.jwksEndpoint).toBe("https://auth.example.com/jwks");
expect(result.oidcConfig.userInfoEndpoint).toBe("https://auth.example.com/userinfo");
expect(result.oidcConfig.scopes).toEqual(["openid", "email", "profile"]);
});
it("throws for invalid issuer URL", async () => {
await expect(
normalizeOidcProviderConfig("not-a-url", {}),
).rejects.toBeInstanceOf(OidcConfigError);
});
});

202
src/lib/sso/oidc-config.ts Normal file
View File

@@ -0,0 +1,202 @@
import { z } from "zod";
const DEFAULT_SCOPES = ["openid", "email", "profile"] as const;
const DISCOVERY_TIMEOUT_MS = 10000;
const discoverySchema = z.object({
issuer: z.string().url().optional(),
authorization_endpoint: z.string().url().optional(),
token_endpoint: z.string().url().optional(),
userinfo_endpoint: z.string().url().optional(),
jwks_uri: z.string().url().optional(),
scopes_supported: z.array(z.string()).optional(),
});
export class OidcConfigError extends Error {
constructor(message: string) {
super(message);
this.name = "OidcConfigError";
}
}
export type RawOidcConfig = {
clientId?: string;
clientSecret?: string;
authorizationEndpoint?: string;
tokenEndpoint?: string;
jwksEndpoint?: string;
userInfoEndpoint?: string;
discoveryEndpoint?: string;
scopes?: string[];
pkce?: boolean;
mapping?: ProviderMapping;
};
export type ProviderMapping = {
id: string;
email: string;
emailVerified?: string;
name?: string;
image?: string;
firstName?: string;
lastName?: string;
};
export type NormalizedOidcConfig = {
oidcConfig: {
clientId?: string;
clientSecret?: string;
authorizationEndpoint: string;
tokenEndpoint: string;
jwksEndpoint?: string;
userInfoEndpoint?: string;
discoveryEndpoint: string;
scopes: string[];
pkce: boolean;
};
mapping: ProviderMapping;
};
type FetchFn = typeof fetch;
function cleanUrl(value: string | undefined, field: string): string | undefined {
if (!value || typeof value !== "string") return undefined;
const trimmed = value.trim();
if (!trimmed) return undefined;
try {
return new URL(trimmed).toString();
} catch {
throw new OidcConfigError(`Invalid ${field} URL: ${value}`);
}
}
function sanitizeScopes(scopes: string[] | undefined, fallback: readonly string[]): string[] {
const candidates = Array.isArray(scopes) ? scopes : [];
const sanitized = candidates
.map(scope => scope?.trim())
.filter((scope): scope is string => Boolean(scope));
if (sanitized.length === 0) {
return [...fallback];
}
return Array.from(new Set(sanitized));
}
async function fetchDiscoveryDocument(url: string, fetchFn: FetchFn): Promise<z.infer<typeof discoverySchema>> {
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), DISCOVERY_TIMEOUT_MS);
try {
const response = await fetchFn(url, {
signal: controller.signal,
headers: { Accept: "application/json" },
});
if (!response.ok) {
throw new OidcConfigError(`OIDC discovery request failed (${response.status} ${response.statusText})`);
}
let payload: unknown;
try {
payload = await response.json();
} catch {
throw new OidcConfigError("OIDC discovery response is not valid JSON");
}
const parsed = discoverySchema.parse(payload);
if (!parsed.authorization_endpoint || !parsed.token_endpoint) {
throw new OidcConfigError("OIDC discovery document is missing required endpoints");
}
return parsed;
} catch (error) {
if (error instanceof OidcConfigError) {
throw error;
}
if (error instanceof Error && error.name === "AbortError") {
throw new OidcConfigError(`OIDC discovery timed out after ${DISCOVERY_TIMEOUT_MS / 1000}s`);
}
throw new OidcConfigError(`Failed to fetch OIDC discovery document: ${error instanceof Error ? error.message : "unknown error"}`);
} finally {
clearTimeout(timeoutId);
}
}
export async function normalizeOidcProviderConfig(
issuer: string,
rawConfig: RawOidcConfig,
fetchFn: FetchFn = fetch,
): Promise<NormalizedOidcConfig> {
if (!issuer || typeof issuer !== "string") {
throw new OidcConfigError("Issuer is required");
}
let normalizedIssuer: string;
try {
const issuerUrl = new URL(issuer.trim());
normalizedIssuer = issuerUrl.toString().replace(/\/$/, "");
} catch {
throw new OidcConfigError(`Invalid issuer URL: ${issuer}`);
}
const discoveryEndpoint = cleanUrl(
rawConfig.discoveryEndpoint,
"discovery endpoint",
) ?? `${normalizedIssuer}/.well-known/openid-configuration`;
const authorizationEndpoint = cleanUrl(rawConfig.authorizationEndpoint, "authorization endpoint");
const tokenEndpoint = cleanUrl(rawConfig.tokenEndpoint, "token endpoint");
const jwksEndpoint = cleanUrl(rawConfig.jwksEndpoint, "JWKS endpoint");
const userInfoEndpoint = cleanUrl(rawConfig.userInfoEndpoint, "userinfo endpoint");
const providedScopes = Array.isArray(rawConfig.scopes) ? rawConfig.scopes : undefined;
let scopes = sanitizeScopes(providedScopes, DEFAULT_SCOPES);
const shouldFetchDiscovery =
!authorizationEndpoint ||
!tokenEndpoint ||
!jwksEndpoint ||
!userInfoEndpoint ||
!providedScopes ||
providedScopes.length === 0;
let resolvedAuthorization = authorizationEndpoint;
let resolvedToken = tokenEndpoint;
let resolvedJwks = jwksEndpoint;
let resolvedUserInfo = userInfoEndpoint;
if (shouldFetchDiscovery) {
const discovery = await fetchDiscoveryDocument(discoveryEndpoint, fetchFn);
resolvedAuthorization = resolvedAuthorization ?? discovery.authorization_endpoint;
resolvedToken = resolvedToken ?? discovery.token_endpoint;
resolvedJwks = resolvedJwks ?? discovery.jwks_uri;
resolvedUserInfo = resolvedUserInfo ?? discovery.userinfo_endpoint;
if (!providedScopes || providedScopes.length === 0) {
scopes = sanitizeScopes(discovery.scopes_supported, DEFAULT_SCOPES);
}
}
if (!resolvedAuthorization || !resolvedToken) {
throw new OidcConfigError("OIDC configuration must include authorization and token endpoints");
}
return {
oidcConfig: {
clientId: rawConfig.clientId,
clientSecret: rawConfig.clientSecret,
authorizationEndpoint: resolvedAuthorization,
tokenEndpoint: resolvedToken,
jwksEndpoint: resolvedJwks,
userInfoEndpoint: resolvedUserInfo,
discoveryEndpoint,
scopes,
pkce: rawConfig.pkce !== false,
},
mapping: rawConfig.mapping ?? {
id: "sub",
email: "email",
emailVerified: "email_verified",
name: "name",
image: "picture",
},
};
}

View File

@@ -29,6 +29,31 @@ export function formatDate(date?: Date | string | null): string {
}).format(new Date(date));
}
export function formatLastSyncTime(date: Date | string | null): string {
if (!date) return "Never";
const now = new Date();
const syncDate = new Date(date);
const diffMs = now.getTime() - syncDate.getTime();
const diffMins = Math.floor(diffMs / 60000);
const diffHours = Math.floor(diffMs / 3600000);
const diffDays = Math.floor(diffMs / 86400000);
// Show relative time for recent syncs
if (diffMins < 1) return "Just now";
if (diffMins < 60) return `${diffMins} min ago`;
if (diffHours < 24) return `${diffHours} hr${diffHours === 1 ? '' : 's'} ago`;
if (diffDays < 7) return `${diffDays} day${diffDays === 1 ? '' : 's'} ago`;
// For older syncs, show week count
const diffWeeks = Math.floor(diffDays / 7);
if (diffWeeks < 4) return `${diffWeeks} week${diffWeeks === 1 ? '' : 's'} ago`;
// For even older, show month count
const diffMonths = Math.floor(diffDays / 30);
return `${diffMonths} month${diffMonths === 1 ? '' : 's'} ago`;
}
export function truncate(str: string, length: number): string {
if (str.length <= length) return str;
return str.slice(0, length) + "...";

View File

@@ -10,7 +10,7 @@
export async function processInParallel<T, R>(
items: T[],
processItem: (item: T) => Promise<R>,
concurrencyLimit: number = 5,
concurrencyLimit: number = 5, // Safe default for GitHub API (max 100 concurrent, but 5-10 recommended)
onProgress?: (completed: number, total: number, result?: R) => void
): Promise<R[]> {
const results: R[] = [];

View File

@@ -93,13 +93,20 @@ export async function createDefaultConfig({ userId, envOverrides = {} }: Default
enabled: scheduleEnabled,
interval: scheduleInterval,
concurrent: false,
batchSize: 10,
batchSize: 5, // Reduced from 10 to be more conservative with GitHub API limits
lastRun: null,
nextRun: scheduleEnabled ? new Date(Date.now() + scheduleInterval * 1000) : null,
},
cleanupConfig: {
enabled: cleanupEnabled,
retentionDays: cleanupRetentionDays,
deleteFromGitea: false,
deleteIfNotInGitHub: true,
protectedRepos: [],
dryRun: false,
orphanedRepoAction: "archive",
batchSize: 10,
pauseBetweenDeletes: 2000,
lastRun: null,
nextRun: cleanupEnabled ? new Date(Date.now() + getCleanupInterval(cleanupRetentionDays) * 1000) : null,
},
@@ -123,4 +130,4 @@ function getCleanupInterval(retentionSeconds: number): number {
if (days <= 7) return 86400; // 24 hours
if (days <= 30) return 172800; // 48 hours
return 604800; // 1 week
}
}

View File

@@ -11,6 +11,7 @@ import type {
} from "@/types/config";
import { z } from "zod";
import { githubConfigSchema, giteaConfigSchema, scheduleConfigSchema, cleanupConfigSchema } from "@/lib/db/schema";
import { parseInterval } from "@/lib/utils/duration-parser";
// Use the actual database schema types
type DbGitHubConfig = z.infer<typeof githubConfigSchema>;
@@ -53,7 +54,7 @@ export function mapUiToDbConfig(
defaultOrg: giteaConfig.organization,
// Advanced options
skipStarredIssues: advancedOptions.skipStarredIssues,
starredCodeOnly: advancedOptions.starredCodeOnly,
};
// Map Gitea config to match database schema
@@ -151,7 +152,8 @@ export function mapDbToUiConfig(dbConfig: any): {
// Map advanced options
const advancedOptions: AdvancedOptions = {
skipForks: !(dbConfig.githubConfig?.includeForks ?? true), // Invert includeForks to get skipForks
skipStarredIssues: dbConfig.githubConfig?.skipStarredIssues || false,
// Support both old (skipStarredIssues) and new (starredCodeOnly) field names for backward compatibility
starredCodeOnly: dbConfig.githubConfig?.starredCodeOnly ?? (dbConfig.githubConfig as any)?.skipStarredIssues ?? false,
};
return {
@@ -165,27 +167,22 @@ export function mapDbToUiConfig(dbConfig: any): {
/**
* Maps UI schedule config to database schema
*/
export function mapUiScheduleToDb(uiSchedule: any): DbScheduleConfig {
export function mapUiScheduleToDb(uiSchedule: any, existing?: DbScheduleConfig): DbScheduleConfig {
// Preserve existing schedule config and only update fields controlled by the UI
const base: DbScheduleConfig = existing
? { ...(existing as unknown as DbScheduleConfig) }
: (scheduleConfigSchema.parse({}) as unknown as DbScheduleConfig);
// Store interval as seconds string to avoid lossy cron conversion
const intervalSeconds = typeof uiSchedule.interval === 'number' && uiSchedule.interval > 0
? String(uiSchedule.interval)
: (typeof base.interval === 'string' ? base.interval : String(86400));
return {
enabled: uiSchedule.enabled || false,
interval: uiSchedule.interval ? `0 */${Math.floor(uiSchedule.interval / 3600)} * * *` : "0 2 * * *", // Convert seconds to cron expression
concurrent: false,
batchSize: 10,
pauseBetweenBatches: 5000,
retryAttempts: 3,
retryDelay: 60000,
timeout: 3600000,
autoRetry: true,
cleanupBeforeMirror: false,
notifyOnFailure: true,
notifyOnSuccess: false,
logLevel: "info",
timezone: "UTC",
onlyMirrorUpdated: false,
updateInterval: 86400000,
skipRecentlyMirrored: true,
recentThreshold: 3600000,
};
...base,
enabled: !!uiSchedule.enabled,
interval: intervalSeconds,
} as DbScheduleConfig;
}
/**
@@ -202,23 +199,18 @@ export function mapDbScheduleToUi(dbSchedule: DbScheduleConfig): any {
};
}
// Extract hours from cron expression if possible
// Parse interval supporting numbers (seconds), duration strings, and cron
let intervalSeconds = 86400; // Default to daily (24 hours)
if (dbSchedule.interval) {
// Check if it's already a number (seconds), use it directly
if (typeof dbSchedule.interval === 'number') {
intervalSeconds = dbSchedule.interval;
} else if (typeof dbSchedule.interval === 'string') {
// Check if it's a cron expression
const cronMatch = dbSchedule.interval.match(/0 \*\/(\d+) \* \* \*/);
if (cronMatch) {
intervalSeconds = parseInt(cronMatch[1]) * 3600;
} else if (dbSchedule.interval === "0 2 * * *") {
// Daily at 2 AM
intervalSeconds = 86400;
}
}
try {
const ms = parseInterval(
typeof dbSchedule.interval === 'number'
? dbSchedule.interval
: (dbSchedule.interval as unknown as string)
);
intervalSeconds = Math.max(1, Math.floor(ms / 1000));
} catch (_e) {
// Fallback to default if unparsable
intervalSeconds = 86400;
}
return {
@@ -233,16 +225,26 @@ export function mapDbScheduleToUi(dbSchedule: DbScheduleConfig): any {
* Maps UI cleanup config to database schema
*/
export function mapUiCleanupToDb(uiCleanup: any): DbCleanupConfig {
const parsedRetention =
typeof uiCleanup.retentionDays === "string"
? parseInt(uiCleanup.retentionDays, 10)
: uiCleanup.retentionDays;
const retentionSeconds = Number.isFinite(parsedRetention)
? parsedRetention
: 604800;
return {
enabled: uiCleanup.enabled || false,
retentionDays: uiCleanup.retentionDays || 604800, // Default to 7 days
deleteFromGitea: false,
deleteIfNotInGitHub: true,
protectedRepos: [],
dryRun: true,
orphanedRepoAction: "archive",
batchSize: 10,
pauseBetweenDeletes: 2000,
enabled: Boolean(uiCleanup.enabled),
retentionDays: retentionSeconds,
deleteFromGitea: uiCleanup.deleteFromGitea ?? false,
deleteIfNotInGitHub: uiCleanup.deleteIfNotInGitHub ?? true,
protectedRepos: uiCleanup.protectedRepos ?? [],
dryRun: uiCleanup.dryRun ?? false,
orphanedRepoAction: (uiCleanup.orphanedRepoAction as DbCleanupConfig["orphanedRepoAction"]) || "archive",
batchSize: uiCleanup.batchSize ?? 10,
pauseBetweenDeletes: uiCleanup.pauseBetweenDeletes ?? 2000,
lastRun: uiCleanup.lastRun ?? null,
nextRun: uiCleanup.nextRun ?? null,
};
}
@@ -261,9 +263,16 @@ export function mapDbCleanupToUi(dbCleanup: DbCleanupConfig): any {
}
return {
enabled: dbCleanup.enabled || false,
retentionDays: dbCleanup.retentionDays || 604800, // Use actual value from DB or default to 7 days
lastRun: dbCleanup.lastRun || null,
nextRun: dbCleanup.nextRun || null,
enabled: dbCleanup.enabled ?? false,
retentionDays: dbCleanup.retentionDays ?? 604800,
deleteFromGitea: dbCleanup.deleteFromGitea ?? false,
deleteIfNotInGitHub: dbCleanup.deleteIfNotInGitHub ?? true,
protectedRepos: dbCleanup.protectedRepos ?? [],
dryRun: dbCleanup.dryRun ?? false,
orphanedRepoAction: dbCleanup.orphanedRepoAction ?? "archive",
batchSize: dbCleanup.batchSize ?? 10,
pauseBetweenDeletes: dbCleanup.pauseBetweenDeletes ?? 2000,
lastRun: dbCleanup.lastRun ?? null,
nextRun: dbCleanup.nextRun ?? null,
};
}
}

View File

@@ -8,6 +8,7 @@ import { setupSignalHandlers } from './lib/signal-handlers';
import { auth } from './lib/auth';
import { isHeaderAuthEnabled, authenticateWithHeaders } from './lib/auth-header';
import { initializeConfigFromEnv } from './lib/env-config-loader';
import { db, users } from './lib/db';
// Flag to track if recovery has been initialized
let recoveryInitialized = false;
@@ -17,6 +18,7 @@ let schedulerServiceStarted = false;
let repositoryCleanupServiceStarted = false;
let shutdownManagerInitialized = false;
let envConfigInitialized = false;
let envConfigCheckCount = 0; // Track attempts to avoid excessive checking
export const onRequest = defineMiddleware(async (context, next) => {
// First, try Better Auth session (cookie-based)
@@ -79,14 +81,31 @@ export const onRequest = defineMiddleware(async (context, next) => {
}
}
// Initialize configuration from environment variables (only once)
if (!envConfigInitialized) {
envConfigInitialized = true;
try {
await initializeConfigFromEnv();
} catch (error) {
console.error('⚠️ Failed to initialize configuration from environment:', error);
// Continue anyway - environment config is optional
// Initialize configuration from environment variables
// Optimized to minimize performance impact:
// - Once initialized, no checks are performed (envConfigInitialized = true)
// - Limits checks to first 100 requests to avoid DB queries on every request if no users exist
// - After user creation, env vars load on next request and flag is set permanently
if (!envConfigInitialized && envConfigCheckCount < 100) {
envConfigCheckCount++;
// Only check every 10th request after the first 10 to reduce DB load
const shouldCheck = envConfigCheckCount <= 10 || envConfigCheckCount % 10 === 0;
if (shouldCheck) {
try {
const hasUsers = await db.select().from(users).limit(1).then(u => u.length > 0);
if (hasUsers) {
// We have users now, try to initialize config
await initializeConfigFromEnv();
envConfigInitialized = true; // This ensures we never check again
console.log('✅ Environment configuration loaded after user creation');
}
} catch (error) {
console.error('⚠️ Failed to initialize configuration from environment:', error);
// Continue anyway - environment config is optional
}
}
}
@@ -160,7 +179,10 @@ export const onRequest = defineMiddleware(async (context, next) => {
if (recoveryInitialized && !schedulerServiceStarted) {
try {
console.log('Starting automatic mirror scheduler service...');
startSchedulerService();
// Start the scheduler service (now async)
startSchedulerService().catch(error => {
console.error('Error in scheduler service startup:', error);
});
// Register scheduler service shutdown callback
registerShutdownCallback(async () => {

View File

@@ -2,6 +2,10 @@ import type { APIContext } from "astro";
import { createSecureErrorResponse } from "@/lib/utils";
import { requireAuth } from "@/lib/utils/auth-helpers";
import { auth } from "@/lib/auth";
import { db, ssoProviders } from "@/lib/db";
import { eq } from "drizzle-orm";
import { nanoid } from "nanoid";
import { normalizeOidcProviderConfig, OidcConfigError } from "@/lib/sso/oidc-config";
// POST /api/auth/sso/register - Register a new SSO provider using Better Auth
export async function POST(context: APIContext) {
@@ -25,9 +29,34 @@ export async function POST(context: APIContext) {
);
}
// Validate issuer URL format
let validatedIssuer = issuer;
if (issuer && typeof issuer === 'string' && issuer.trim() !== '') {
try {
const issuerUrl = new URL(issuer.trim());
validatedIssuer = issuerUrl.toString().replace(/\/$/, ''); // Remove trailing slash
} catch (e) {
return new Response(
JSON.stringify({ error: `Invalid issuer URL format: ${issuer}` }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
} else {
return new Response(
JSON.stringify({ error: "Issuer URL cannot be empty" }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
let registrationBody: any = {
providerId,
issuer,
issuer: validatedIssuer,
domain,
organizationId,
};
@@ -79,30 +108,37 @@ export async function POST(context: APIContext) {
userInfoEndpoint,
scopes,
pkce = true,
mapping = {
id: "sub",
email: "email",
emailVerified: "email_verified",
name: "name",
image: "picture",
}
mapping,
} = body;
// Use provided scopes or default if not specified
const finalScopes = scopes || ["openid", "email", "profile"];
try {
const normalized = await normalizeOidcProviderConfig(validatedIssuer, {
clientId,
clientSecret,
authorizationEndpoint,
tokenEndpoint,
jwksEndpoint,
userInfoEndpoint,
discoveryEndpoint,
scopes,
pkce,
mapping,
});
registrationBody.oidcConfig = {
clientId,
clientSecret,
authorizationEndpoint,
tokenEndpoint,
jwksEndpoint,
discoveryEndpoint,
userInfoEndpoint,
scopes: finalScopes,
pkce,
};
registrationBody.mapping = mapping;
registrationBody.oidcConfig = normalized.oidcConfig;
registrationBody.mapping = normalized.mapping;
} catch (error) {
if (error instanceof OidcConfigError) {
return new Response(
JSON.stringify({ error: error.message }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
throw error;
}
}
// Get the user's auth headers to make the request
@@ -130,7 +166,52 @@ export async function POST(context: APIContext) {
}
const result = await response.json();
// Mirror provider entry into local SSO table for UI listing
try {
const existing = await db
.select()
.from(ssoProviders)
.where(eq(ssoProviders.providerId, registrationBody.providerId))
.limit(1);
const values: any = {
issuer: registrationBody.issuer,
domain: registrationBody.domain,
organizationId: registrationBody.organizationId,
updatedAt: new Date(),
};
if (registrationBody.oidcConfig) {
values.oidcConfig = JSON.stringify({
...registrationBody.oidcConfig,
mapping: registrationBody.mapping,
});
}
if (existing.length > 0) {
await db
.update(ssoProviders)
.set(values)
.where(eq(ssoProviders.id, existing[0].id));
} else {
await db.insert(ssoProviders).values({
id: nanoid(),
issuer: registrationBody.issuer,
domain: registrationBody.domain,
oidcConfig: JSON.stringify({
...registrationBody.oidcConfig,
mapping: registrationBody.mapping,
}),
userId: user.id,
providerId: registrationBody.providerId,
organizationId: registrationBody.organizationId,
});
}
} catch (mirroringError) {
console.warn("Failed to mirror SSO provider to local DB:", mirroringError);
}
return new Response(JSON.stringify(result), {
status: 201,
headers: { "Content-Type": "application/json" },
@@ -161,4 +242,4 @@ export async function GET(context: APIContext) {
} catch (error) {
return createSecureErrorResponse(error, "SSO provider listing");
}
}
}

View File

@@ -87,7 +87,10 @@ export const POST: APIRoute = async ({ request }) => {
}
// Map schedule and cleanup configs to database schema
const processedScheduleConfig = mapUiScheduleToDb(scheduleConfig);
const processedScheduleConfig = mapUiScheduleToDb(
scheduleConfig,
existingConfig ? existingConfig.scheduleConfig : undefined
);
const processedCleanupConfig = mapUiCleanupToDb(cleanupConfig);
if (existingConfig) {

View File

@@ -0,0 +1,69 @@
import type { APIRoute } from "astro";
import { getNewEvents } from "@/lib/events";
export const GET: APIRoute = async ({ request }) => {
const url = new URL(request.url);
const userId = url.searchParams.get("userId");
if (!userId) {
return new Response("Missing userId", { status: 400 });
}
// Create a new ReadableStream for SSE
const stream = new ReadableStream({
start(controller) {
const encoder = new TextEncoder();
let lastEventTime = new Date();
// Send initial connection message
controller.enqueue(encoder.encode(": connected\n\n"));
// Poll for new events every 2 seconds
const pollInterval = setInterval(async () => {
try {
// Get new rate limit events
const newEvents = await getNewEvents({
userId,
channel: "rate-limit",
lastEventTime,
});
// Send each new event
for (const event of newEvents) {
const message = `event: rate-limit\ndata: ${JSON.stringify(event.payload)}\n\n`;
controller.enqueue(encoder.encode(message));
lastEventTime = new Date(event.createdAt);
}
} catch (error) {
console.error("Error polling for events:", error);
}
}, 2000); // Poll every 2 seconds
// Send heartbeat every 30 seconds to keep connection alive
const heartbeatInterval = setInterval(() => {
try {
controller.enqueue(encoder.encode(": heartbeat\n\n"));
} catch (error) {
clearInterval(heartbeatInterval);
clearInterval(pollInterval);
}
}, 30000);
// Cleanup on close
request.signal.addEventListener("abort", () => {
clearInterval(pollInterval);
clearInterval(heartbeatInterval);
controller.close();
});
},
});
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no", // Disable nginx buffering
},
});
};

View File

@@ -71,9 +71,10 @@ export const POST: APIRoute = async ({ request }) => {
throw new Error("GitHub token is missing in config.");
}
// Create a single Octokit instance to be reused
// Create a single Octokit instance to be reused with rate limit tracking
const decryptedToken = getDecryptedGitHubToken(config);
const octokit = createGitHubClient(decryptedToken);
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedToken, userId, githubUsername);
// Define the concurrency limit - adjust based on API rate limits
// Using a lower concurrency for organizations since each org might contain many repos

View File

@@ -73,9 +73,10 @@ export const POST: APIRoute = async ({ request }) => {
throw new Error("GitHub token is missing.");
}
// Create a single Octokit instance to be reused
// Create a single Octokit instance to be reused with rate limit tracking
const decryptedToken = getDecryptedGitHubToken(config);
const octokit = createGitHubClient(decryptedToken);
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedToken, userId, githubUsername);
// Define the concurrency limit - adjust based on API rate limits
const CONCURRENCY_LIMIT = 3;

View File

@@ -71,12 +71,13 @@ export const POST: APIRoute = async ({ request }) => {
// Start background retry with parallel processing
setTimeout(async () => {
// Create a single Octokit instance to be reused if needed
// Create a single Octokit instance to be reused if needed with rate limit tracking
const decryptedToken = config.githubConfig.token
? getDecryptedGitHubToken(config)
: null;
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = decryptedToken
? createGitHubClient(decryptedToken)
? createGitHubClient(decryptedToken, userId, githubUsername)
: null;
// Define the concurrency limit - adjust based on API rate limits

View File

@@ -8,6 +8,7 @@ import type {
ScheduleSyncRepoResponse,
} from "@/types/sync";
import { createSecureErrorResponse } from "@/lib/utils";
import { parseInterval } from "@/lib/utils/duration-parser";
export const POST: APIRoute = async ({ request }) => {
try {
@@ -72,8 +73,17 @@ export const POST: APIRoute = async ({ request }) => {
// Calculate nextRun and update lastRun and nextRun in the config
const currentTime = new Date();
const interval = config.scheduleConfig?.interval ?? 3600;
const nextRun = new Date(currentTime.getTime() + interval * 1000);
let intervalMs = 3600 * 1000;
try {
intervalMs = parseInterval(
typeof config.scheduleConfig?.interval === 'number'
? config.scheduleConfig.interval
: (config.scheduleConfig?.interval as unknown as string) || '3600'
);
} catch {
intervalMs = 3600 * 1000;
}
const nextRun = new Date(currentTime.getTime() + intervalMs);
// Update the full giteaConfig object
await db

View File

@@ -0,0 +1,104 @@
import type { APIRoute } from "astro";
import { db, rateLimits } from "@/lib/db";
import { eq, and, desc } from "drizzle-orm";
import { jsonResponse, createSecureErrorResponse } from "@/lib/utils";
import { RateLimitManager } from "@/lib/rate-limit-manager";
import { createGitHubClient } from "@/lib/github";
import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption";
import { configs } from "@/lib/db";
export const GET: APIRoute = async ({ request }) => {
const url = new URL(request.url);
const userId = url.searchParams.get("userId");
const refresh = url.searchParams.get("refresh") === "true";
if (!userId) {
return jsonResponse({
data: { error: "Missing userId" },
status: 400,
});
}
try {
// If refresh is requested, fetch current rate limit from GitHub
if (refresh) {
const [config] = await db
.select()
.from(configs)
.where(eq(configs.userId, userId))
.limit(1);
if (config && config.githubConfig?.token) {
const decryptedToken = getDecryptedGitHubToken(config);
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedToken, userId, githubUsername);
// This will update the rate limit in the database
await RateLimitManager.checkGitHubRateLimit(octokit, userId);
}
}
// Get rate limit status from database
const [rateLimit] = await db
.select()
.from(rateLimits)
.where(and(eq(rateLimits.userId, userId), eq(rateLimits.provider, "github")))
.orderBy(desc(rateLimits.updatedAt))
.limit(1);
if (!rateLimit) {
return jsonResponse({
data: {
limit: 5000,
remaining: 5000,
used: 0,
reset: new Date(Date.now() + 3600000), // 1 hour from now
status: "ok",
lastChecked: new Date(),
message: "No rate limit data available yet",
},
});
}
// Calculate percentage
const percentage = Math.round((rateLimit.remaining / rateLimit.limit) * 100);
// Calculate time until reset
const now = new Date();
const resetTime = new Date(rateLimit.reset);
const timeUntilReset = Math.max(0, resetTime.getTime() - now.getTime());
const minutesUntilReset = Math.ceil(timeUntilReset / 60000);
let message = "";
switch (rateLimit.status) {
case "exceeded":
message = `Rate limit exceeded. Resets in ${minutesUntilReset} minutes.`;
break;
case "limited":
message = `Rate limit critical: ${rateLimit.remaining}/${rateLimit.limit} (${percentage}%)`;
break;
case "warning":
message = `Rate limit warning: ${rateLimit.remaining}/${rateLimit.limit} (${percentage}%)`;
break;
default:
message = `Rate limit healthy: ${rateLimit.remaining}/${rateLimit.limit} (${percentage}%)`;
}
return jsonResponse({
data: {
limit: rateLimit.limit,
remaining: rateLimit.remaining,
used: rateLimit.used,
reset: rateLimit.reset,
retryAfter: rateLimit.retryAfter,
status: rateLimit.status,
lastChecked: rateLimit.lastChecked,
percentage,
minutesUntilReset,
message,
},
});
} catch (error) {
return createSecureErrorResponse(error, "rate limit check", 500);
}
};

View File

@@ -10,26 +10,71 @@ export async function POST(context: APIContext) {
const { issuer } = await context.request.json();
if (!issuer) {
return new Response(JSON.stringify({ error: "Issuer URL is required" }), {
if (!issuer || typeof issuer !== 'string' || issuer.trim() === '') {
return new Response(JSON.stringify({ error: "Issuer URL is required and must be a valid string" }), {
status: 400,
headers: { "Content-Type": "application/json" },
});
}
// Ensure issuer URL ends without trailing slash for well-known discovery
const cleanIssuer = issuer.replace(/\/$/, "");
// Validate issuer URL format
let cleanIssuer: string;
try {
const issuerUrl = new URL(issuer.trim());
cleanIssuer = issuerUrl.toString().replace(/\/$/, ""); // Remove trailing slash
} catch (e) {
return new Response(
JSON.stringify({
error: "Invalid issuer URL format",
details: `The provided URL "${issuer}" is not a valid URL. For Authentik, use format: https://your-authentik-domain/application/o/<app-slug>/`
}),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
const discoveryUrl = `${cleanIssuer}/.well-known/openid-configuration`;
try {
// Fetch OIDC discovery document
const response = await fetch(discoveryUrl);
// Fetch OIDC discovery document with timeout
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 10000); // 10 second timeout
let response: Response;
try {
response = await fetch(discoveryUrl, {
signal: controller.signal,
headers: {
'Accept': 'application/json',
}
});
} catch (fetchError) {
if (fetchError instanceof Error && fetchError.name === 'AbortError') {
throw new Error(`Request timeout: The OIDC provider at ${cleanIssuer} did not respond within 10 seconds`);
}
throw new Error(`Network error: Could not connect to ${cleanIssuer}. Please verify the URL is correct and accessible.`);
} finally {
clearTimeout(timeoutId);
}
if (!response.ok) {
throw new Error(`Failed to fetch discovery document: ${response.status}`);
if (response.status === 404) {
throw new Error(`OIDC discovery document not found at ${discoveryUrl}. For Authentik, ensure you're using the correct application slug in the URL.`);
} else if (response.status >= 500) {
throw new Error(`OIDC provider error (${response.status}): The server at ${cleanIssuer} returned an error.`);
} else {
throw new Error(`Failed to fetch discovery document (${response.status}): ${response.statusText}`);
}
}
const config = await response.json();
let config: any;
try {
config = await response.json();
} catch (parseError) {
throw new Error(`Invalid response: The discovery document from ${cleanIssuer} is not valid JSON.`);
}
// Extract the essential endpoints
const discoveredConfig = {

View File

@@ -4,6 +4,7 @@ import { requireAuth } from "@/lib/utils/auth-helpers";
import { db, ssoProviders } from "@/lib/db";
import { nanoid } from "nanoid";
import { eq } from "drizzle-orm";
import { normalizeOidcProviderConfig, OidcConfigError, type RawOidcConfig } from "@/lib/sso/oidc-config";
// GET /api/sso/providers - List all SSO providers
export async function GET(context: APIContext) {
@@ -45,10 +46,12 @@ export async function POST(context: APIContext) {
tokenEndpoint,
jwksEndpoint,
userInfoEndpoint,
discoveryEndpoint,
mapping,
providerId,
organizationId,
scopes,
pkce,
} = body;
// Validate required fields
@@ -79,22 +82,51 @@ export async function POST(context: APIContext) {
);
}
// Create OIDC config object
const oidcConfig = {
clientId,
clientSecret,
authorizationEndpoint,
tokenEndpoint,
jwksEndpoint,
userInfoEndpoint,
scopes: scopes || ["openid", "email", "profile"],
mapping: mapping || {
id: "sub",
email: "email",
emailVerified: "email_verified",
name: "name",
image: "picture",
},
// Clean issuer URL (remove trailing slash); validate format
let cleanIssuer = issuer;
try {
const issuerUrl = new URL(issuer.toString().trim());
cleanIssuer = issuerUrl.toString().replace(/\/$/, "");
} catch {
return new Response(
JSON.stringify({ error: `Invalid issuer URL format: ${issuer}` }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
let normalized;
try {
normalized = await normalizeOidcProviderConfig(cleanIssuer, {
clientId,
clientSecret,
authorizationEndpoint,
tokenEndpoint,
jwksEndpoint,
userInfoEndpoint,
discoveryEndpoint,
scopes,
pkce,
mapping,
});
} catch (error) {
if (error instanceof OidcConfigError) {
return new Response(
JSON.stringify({ error: error.message }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
throw error;
}
const storedOidcConfig = {
...normalized.oidcConfig,
mapping: normalized.mapping,
};
// Insert new provider
@@ -102,9 +134,9 @@ export async function POST(context: APIContext) {
.insert(ssoProviders)
.values({
id: nanoid(),
issuer,
issuer: cleanIssuer,
domain,
oidcConfig: JSON.stringify(oidcConfig),
oidcConfig: JSON.stringify(storedOidcConfig),
userId: user.id,
providerId,
organizationId,
@@ -156,7 +188,9 @@ export async function PUT(context: APIContext) {
tokenEndpoint,
jwksEndpoint,
userInfoEndpoint,
discoveryEndpoint,
scopes,
pkce,
organizationId,
} = body;
@@ -179,26 +213,62 @@ export async function PUT(context: APIContext) {
// Parse existing config
const existingConfig = JSON.parse(existingProvider.oidcConfig);
const effectiveIssuer = issuer || existingProvider.issuer;
// Create updated OIDC config
const updatedOidcConfig = {
...existingConfig,
clientId: clientId || existingConfig.clientId,
clientSecret: clientSecret || existingConfig.clientSecret,
authorizationEndpoint: authorizationEndpoint || existingConfig.authorizationEndpoint,
tokenEndpoint: tokenEndpoint || existingConfig.tokenEndpoint,
jwksEndpoint: jwksEndpoint || existingConfig.jwksEndpoint,
userInfoEndpoint: userInfoEndpoint || existingConfig.userInfoEndpoint,
scopes: scopes || existingConfig.scopes || ["openid", "email", "profile"],
let cleanIssuer = effectiveIssuer;
try {
const issuerUrl = new URL(effectiveIssuer.toString().trim());
cleanIssuer = issuerUrl.toString().replace(/\/$/, "");
} catch {
return new Response(
JSON.stringify({ error: `Invalid issuer URL format: ${effectiveIssuer}` }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
const mergedConfig: RawOidcConfig = {
clientId: clientId ?? existingConfig.clientId,
clientSecret: clientSecret ?? existingConfig.clientSecret,
authorizationEndpoint: authorizationEndpoint ?? existingConfig.authorizationEndpoint,
tokenEndpoint: tokenEndpoint ?? existingConfig.tokenEndpoint,
jwksEndpoint: jwksEndpoint ?? existingConfig.jwksEndpoint,
userInfoEndpoint: userInfoEndpoint ?? existingConfig.userInfoEndpoint,
discoveryEndpoint: discoveryEndpoint ?? existingConfig.discoveryEndpoint,
scopes: scopes ?? existingConfig.scopes,
pkce: pkce ?? existingConfig.pkce,
mapping: existingConfig.mapping,
};
let normalized;
try {
normalized = await normalizeOidcProviderConfig(cleanIssuer, mergedConfig);
} catch (error) {
if (error instanceof OidcConfigError) {
return new Response(
JSON.stringify({ error: error.message }),
{
status: 400,
headers: { "Content-Type": "application/json" },
}
);
}
throw error;
}
const storedOidcConfig = {
...normalized.oidcConfig,
mapping: normalized.mapping,
};
// Update provider
const [updatedProvider] = await db
.update(ssoProviders)
.set({
issuer: issuer || existingProvider.issuer,
issuer: cleanIssuer,
domain: domain || existingProvider.domain,
oidcConfig: JSON.stringify(updatedOidcConfig),
oidcConfig: JSON.stringify(storedOidcConfig),
organizationId: organizationId !== undefined ? organizationId : existingProvider.organizationId,
updatedAt: new Date(),
})
@@ -259,4 +329,4 @@ export async function DELETE(context: APIContext) {
} catch (error) {
return createSecureErrorResponse(error, "SSO providers API");
}
}
}

View File

@@ -10,6 +10,7 @@ import {
getGithubStarredRepositories,
} from "@/lib/github";
import { jsonResponse, createSecureErrorResponse } from "@/lib/utils";
import { mergeGitReposPreferStarred, calcBatchSizeForInsert } from "@/lib/repo-utils";
import { getDecryptedGitHubToken } from "@/lib/utils/config-encryption";
export const POST: APIRoute = async ({ request }) => {
@@ -43,7 +44,8 @@ export const POST: APIRoute = async ({ request }) => {
// Decrypt the GitHub token before using it
const decryptedToken = getDecryptedGitHubToken(config);
const octokit = createGitHubClient(decryptedToken);
const githubUsername = config.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedToken, userId, githubUsername);
// Fetch GitHub data in parallel
const [basicAndForkedRepos, starredRepos, gitOrgs] = await Promise.all([
@@ -54,7 +56,8 @@ export const POST: APIRoute = async ({ request }) => {
getGithubOrganizations({ octokit, config }),
]);
const allGithubRepos = [...basicAndForkedRepos, ...starredRepos];
// Merge and de-duplicate by fullName, preferring starred variant when duplicated
const allGithubRepos = mergeGitReposPreferStarred(basicAndForkedRepos, starredRepos);
// Prepare full list of repos and orgs
const newRepos = allGithubRepos.map((repo) => ({
@@ -66,21 +69,25 @@ export const POST: APIRoute = async ({ request }) => {
url: repo.url,
cloneUrl: repo.cloneUrl,
owner: repo.owner,
organization: repo.organization,
organization: repo.organization ?? null,
mirroredLocation: repo.mirroredLocation || "",
destinationOrg: repo.destinationOrg || null,
isPrivate: repo.isPrivate,
isForked: repo.isForked,
forkedFrom: repo.forkedFrom,
forkedFrom: repo.forkedFrom ?? null,
hasIssues: repo.hasIssues,
isStarred: repo.isStarred,
isArchived: repo.isArchived,
size: repo.size,
hasLFS: repo.hasLFS,
hasSubmodules: repo.hasSubmodules,
language: repo.language ?? null,
description: repo.description ?? null,
defaultBranch: repo.defaultBranch,
visibility: repo.visibility,
status: repo.status,
lastMirrored: repo.lastMirrored,
errorMessage: repo.errorMessage,
lastMirrored: repo.lastMirrored ?? null,
errorMessage: repo.errorMessage ?? null,
createdAt: repo.createdAt,
updatedAt: repo.updatedAt,
}));
@@ -123,12 +130,27 @@ export const POST: APIRoute = async ({ request }) => {
);
insertedOrgs = newOrgs.filter((o) => !existingOrgNames.has(o.name));
// Batch insert repositories to avoid SQLite parameter limit (dynamic by column count)
const sample = newRepos[0];
const columnCount = Object.keys(sample ?? {}).length || 1;
const REPO_BATCH_SIZE = calcBatchSizeForInsert(columnCount);
if (insertedRepos.length > 0) {
await tx.insert(repositories).values(insertedRepos);
for (let i = 0; i < insertedRepos.length; i += REPO_BATCH_SIZE) {
const batch = insertedRepos.slice(i, i + REPO_BATCH_SIZE);
await tx
.insert(repositories)
.values(batch)
.onConflictDoNothing({ target: [repositories.userId, repositories.fullName] });
}
}
// Batch insert organizations (they have fewer fields, so we can use larger batches)
const ORG_BATCH_SIZE = 100;
if (insertedOrgs.length > 0) {
await tx.insert(organizations).values(insertedOrgs);
for (let i = 0; i < insertedOrgs.length; i += ORG_BATCH_SIZE) {
const batch = insertedOrgs.slice(i, i + ORG_BATCH_SIZE);
await tx.insert(organizations).values(batch);
}
}
});

View File

@@ -69,8 +69,9 @@ export const POST: APIRoute = async ({ request }) => {
});
}
// Create authenticated Octokit instance
const octokit = createGitHubClient(decryptedConfig.githubConfig.token);
// Create authenticated Octokit instance with rate limit tracking
const githubUsername = decryptedConfig.githubConfig?.owner || undefined;
const octokit = createGitHubClient(decryptedConfig.githubConfig.token, userId, githubUsername);
// Fetch org metadata
const { data: orgData } = await octokit.orgs.get({ org });
@@ -117,25 +118,40 @@ export const POST: APIRoute = async ({ request }) => {
owner: repo.owner.login,
organization:
repo.owner.type === "Organization" ? repo.owner.login : null,
mirroredLocation: "",
destinationOrg: null,
isPrivate: repo.private,
isForked: repo.fork,
forkedFrom: undefined,
forkedFrom: null,
hasIssues: repo.has_issues,
isStarred: false,
isArchived: repo.archived,
size: repo.size,
hasLFS: false,
hasSubmodules: false,
language: repo.language ?? null,
description: repo.description ?? null,
defaultBranch: repo.default_branch ?? "main",
visibility: (repo.visibility ?? "public") as RepositoryVisibility,
status: "imported" as RepoStatus,
lastMirrored: undefined,
errorMessage: undefined,
lastMirrored: null,
errorMessage: null,
createdAt: repo.created_at ? new Date(repo.created_at) : new Date(),
updatedAt: repo.updated_at ? new Date(repo.updated_at) : new Date(),
}));
await db.insert(repositories).values(repoRecords);
// Batch insert repositories to avoid SQLite parameter limit
// Compute batch size based on column count
const sample = repoRecords[0];
const columnCount = Object.keys(sample ?? {}).length || 1;
const BATCH_SIZE = Math.max(1, Math.floor(999 / columnCount));
for (let i = 0; i < repoRecords.length; i += BATCH_SIZE) {
const batch = repoRecords.slice(i, i + BATCH_SIZE);
await db
.insert(repositories)
.values(batch)
.onConflictDoNothing({ target: [repositories.userId, repositories.fullName] });
}
// Insert organization metadata
const organizationRecord = {

View File

@@ -80,24 +80,25 @@ export const POST: APIRoute = async ({ request }) => {
cloneUrl: repoData.clone_url,
owner: repoData.owner.login,
organization:
repoData.owner.type === "Organization"
? repoData.owner.login
: undefined,
repoData.owner.type === "Organization" ? repoData.owner.login : null,
isPrivate: repoData.private,
isForked: repoData.fork,
forkedFrom: undefined,
forkedFrom: null,
hasIssues: repoData.has_issues,
isStarred: false,
isArchived: repoData.archived,
size: repoData.size,
hasLFS: false,
hasSubmodules: false,
language: repoData.language ?? null,
description: repoData.description ?? null,
defaultBranch: repoData.default_branch,
visibility: (repoData.visibility ?? "public") as RepositoryVisibility,
status: "imported" as Repository["status"],
lastMirrored: undefined,
errorMessage: undefined,
lastMirrored: null,
errorMessage: null,
mirroredLocation: "",
destinationOrg: null,
createdAt: repoData.created_at
? new Date(repoData.created_at)
: new Date(),
@@ -106,7 +107,10 @@ export const POST: APIRoute = async ({ request }) => {
: new Date(),
};
await db.insert(repositories).values(metadata);
await db
.insert(repositories)
.values(metadata)
.onConflictDoNothing({ target: [repositories.userId, repositories.fullName] });
createMirrorJob({
userId,

View File

@@ -12,6 +12,7 @@ export const repoStatusEnum = z.enum([
"deleted",
"syncing",
"synced",
"archived",
]);
export type RepoStatus = z.infer<typeof repoStatusEnum>;
@@ -48,6 +49,8 @@ export interface GitRepo {
owner: string;
organization?: string;
mirroredLocation?: string;
destinationOrg?: string | null;
isPrivate: boolean;
isForked: boolean;
@@ -61,6 +64,8 @@ export interface GitRepo {
hasLFS: boolean;
hasSubmodules: boolean;
language?: string | null;
description?: string | null;
defaultBranch: string;
visibility: RepositoryVisibility;

View File

@@ -25,15 +25,25 @@ export interface ScheduleConfig {
export interface DatabaseCleanupConfig {
enabled: boolean;
retentionDays: number; // Actually stores seconds, but keeping the name for compatibility
deleteIfNotInGitHub: boolean;
orphanedRepoAction: "skip" | "archive" | "delete";
dryRun: boolean;
deleteFromGitea?: boolean;
protectedRepos?: string[];
batchSize?: number;
pauseBetweenDeletes?: number;
lastRun?: Date;
nextRun?: Date;
}
export type DuplicateNameStrategy = "suffix" | "prefix" | "owner-org";
export interface GitHubConfig {
username: string;
token: string;
privateRepositories: boolean;
mirrorStarred: boolean;
starredDuplicateStrategy?: DuplicateNameStrategy;
}
export interface MirrorOptions {
@@ -52,7 +62,7 @@ export interface MirrorOptions {
export interface AdvancedOptions {
skipForks: boolean;
skipStarredIssues: boolean;
starredCodeOnly: boolean;
}
export interface SaveConfigApiRequest {

View File

@@ -5,6 +5,7 @@ import type { RepoStatus } from "./Repository";
export const membershipRoleEnum = z.enum([
"member",
"admin",
"owner",
"billing_manager",
]);

View File

@@ -0,0 +1,10 @@
import { describe, expect, it } from "bun:test";
import { repoStatusEnum } from "@/types/Repository";
describe("repoStatusEnum", () => {
it("includes archived status", () => {
const res = repoStatusEnum.safeParse("archived");
expect(res.success).toBe(true);
});
});

View File

@@ -9,28 +9,28 @@
"astro": "astro"
},
"dependencies": {
"@astrojs/mdx": "^4.3.4",
"@astrojs/react": "^4.3.0",
"@astrojs/mdx": "^4.3.6",
"@astrojs/react": "^4.4.0",
"@radix-ui/react-icons": "^1.3.2",
"@radix-ui/react-slot": "^1.2.3",
"@splinetool/react-spline": "^4.1.0",
"@splinetool/runtime": "^1.10.52",
"@tailwindcss/vite": "^4.1.12",
"@splinetool/runtime": "^1.10.73",
"@tailwindcss/vite": "^4.1.14",
"@types/canvas-confetti": "^1.9.0",
"@types/react": "^19.1.12",
"@types/react-dom": "^19.1.9",
"astro": "^5.13.4",
"@types/react": "^19.2.0",
"@types/react-dom": "^19.2.0",
"astro": "^5.14.3",
"canvas-confetti": "^1.9.3",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.542.0",
"react": "^19.1.1",
"react-dom": "^19.1.1",
"lucide-react": "^0.544.0",
"react": "^19.2.0",
"react-dom": "^19.2.0",
"tailwind-merge": "^3.3.1",
"tailwindcss": "^4.1.12"
"tailwindcss": "^4.1.14"
},
"devDependencies": {
"tw-animate-css": "^1.3.7"
"tw-animate-css": "^1.4.0"
},
"packageManager": "pnpm@10.15.0"
"packageManager": "pnpm@10.18.0"
}

Some files were not shown because too many files have changed in this diff Show More