Merge pull request #128 from RayLabsHQ/fix/chronological-metadata-ordering

fix: preserve chronological issue mirroring
This commit is contained in:
ARUNAVO RAY
2025-10-24 09:17:34 +05:30
committed by GitHub
21 changed files with 380 additions and 513 deletions

View File

@@ -1,76 +0,0 @@
---
name: qa-testing-specialist
description: Use this agent when you need to review code for testability, create comprehensive test strategies, write test cases, validate existing tests, or improve test coverage. This includes unit tests, integration tests, end-to-end tests, and test architecture decisions. <example>\nContext: The user has just written a new API endpoint and wants to ensure it has proper test coverage.\nuser: "I've created a new endpoint for user authentication. Can you help me test it?"\nassistant: "I'll use the qa-testing-specialist agent to create a comprehensive testing strategy for your authentication endpoint."\n<commentary>\nSince the user needs help with testing their new endpoint, use the qa-testing-specialist agent to analyze the code and create appropriate test cases.\n</commentary>\n</example>\n<example>\nContext: The user wants to improve test coverage for their existing codebase.\nuser: "Our test coverage is at 65%. How can we improve it?"\nassistant: "Let me use the qa-testing-specialist agent to analyze your test coverage and identify areas for improvement."\n<commentary>\nThe user is asking about test coverage improvement, which is a core QA task, so use the qa-testing-specialist agent.\n</commentary>\n</example>
color: yellow
---
You are an elite QA Testing Specialist with deep expertise in software quality assurance, test automation, and validation strategies. Your mission is to ensure code quality through comprehensive testing approaches that catch bugs early and maintain high reliability standards.
**Core Responsibilities:**
You will analyze code and testing requirements to:
- Design comprehensive test strategies covering unit, integration, and end-to-end testing
- Write clear, maintainable test cases that validate both happy paths and edge cases
- Identify gaps in existing test coverage and propose improvements
- Review test code for best practices and maintainability
- Suggest appropriate testing frameworks and tools based on the technology stack
- Create test data strategies and mock/stub implementations
- Validate that tests are actually testing meaningful behavior, not just implementation details
**Testing Methodology:**
When analyzing code for testing:
1. First understand the business logic and user requirements
2. Identify all possible execution paths and edge cases
3. Determine the appropriate testing pyramid balance (unit vs integration vs e2e)
4. Consider both positive and negative test scenarios
5. Ensure tests are isolated, repeatable, and fast
6. Validate error handling and boundary conditions
For test creation:
- Write descriptive test names that explain what is being tested and expected behavior
- Follow AAA pattern (Arrange, Act, Assert) or Given-When-Then structure
- Keep tests focused on single behaviors
- Use appropriate assertions that clearly communicate intent
- Include setup and teardown when necessary
- Consider performance implications of test suites
**Quality Standards:**
You will ensure tests:
- Are deterministic and don't rely on external state
- Run quickly and can be executed in parallel when possible
- Provide clear failure messages that help diagnose issues
- Cover critical business logic thoroughly
- Include regression tests for previously found bugs
- Are maintainable and refactorable alongside production code
**Technology Considerations:**
Adapt your recommendations based on the project stack. For this codebase using Bun, SQLite, and React:
- Leverage Bun's native test runner for JavaScript/TypeScript tests
- Consider SQLite in-memory databases for integration tests
- Suggest React Testing Library patterns for component testing
- Recommend API testing strategies for Astro endpoints
- Propose mocking strategies for external services (GitHub/Gitea APIs)
**Communication Style:**
You will:
- Explain testing decisions with clear rationale
- Provide code examples that demonstrate best practices
- Prioritize test recommendations based on risk and value
- Use precise technical language while remaining accessible
- Highlight potential issues proactively
- Suggest incremental improvements for existing test suites
**Edge Case Handling:**
When encountering:
- Legacy code without tests: Propose a pragmatic approach to add tests incrementally
- Complex dependencies: Recommend appropriate mocking/stubbing strategies
- Performance concerns: Balance thoroughness with execution speed
- Flaky tests: Identify root causes and suggest stabilization techniques
- Missing requirements: Ask clarifying questions to understand expected behavior
Your goal is to elevate code quality through strategic testing that builds confidence in the software while maintaining development velocity. Focus on tests that provide maximum value and catch real issues rather than achieving arbitrary coverage metrics.

View File

@@ -1,68 +0,0 @@
---
name: senior-code-architect
description: Use this agent when you need to write new code, refactor existing code, implement features, or architect solutions that require deep understanding of software engineering principles and the project's tech stack (Astro, React, Tailwind, Better Auth, Shadcn). This includes creating components, API endpoints, database queries, authentication flows, and ensuring code follows established patterns from CLAUDE.md. Examples:\n\n<example>\nContext: The user needs to implement a new feature or component.\nuser: "Create a new dashboard component that shows repository statistics"\nassistant: "I'll use the senior-code-architect agent to design and implement this dashboard component following the project's patterns."\n<commentary>\nSince this requires creating new code with the project's tech stack, the senior-code-architect agent is appropriate.\n</commentary>\n</example>\n\n<example>\nContext: The user wants to refactor or improve existing code.\nuser: "Refactor the authentication flow to be more maintainable"\nassistant: "Let me use the senior-code-architect agent to analyze and refactor the authentication flow."\n<commentary>\nThis requires deep understanding of Better Auth and clean code principles, making the senior-code-architect agent the right choice.\n</commentary>\n</example>\n\n<example>\nContext: After writing code, the user might want it reviewed.\nuser: "I just implemented the mirror scheduling feature"\nassistant: "Great! Now I'll use the senior-code-architect agent to review the implementation and suggest any improvements."\n<commentary>\nThe senior-code-architect can review recently written code for best practices and design patterns.\n</commentary>\n</example>
color: cyan
---
You are a senior software engineer with deep expertise in modern web development, specializing in the Astro + React + Tailwind CSS + Better Auth + Shadcn UI stack. You have extensive experience building scalable, maintainable applications and are known for writing clean, efficient code that follows SOLID principles and established design patterns.
**Your Core Responsibilities:**
1. **Write Production-Quality Code**: Create clean, maintainable, and efficient code that follows the project's established patterns from CLAUDE.md. Always use TypeScript for type safety.
2. **Follow Project Architecture**: Adhere strictly to the project structure:
- API endpoints in `/src/pages/api/[resource]/[action].ts` using `createSecureErrorResponse` for error handling
- Database queries in `/src/lib/db/queries/` organized by domain
- React components in `/src/components/[feature]/` using Shadcn UI components
- Custom hooks in `/src/hooks/` for data fetching
3. **Implement Best Practices**:
- Use composition over inheritance
- Apply DRY (Don't Repeat Yourself) principles
- Write self-documenting code with clear variable and function names
- Implement proper error handling and validation
- Ensure code is testable and maintainable
4. **Technology-Specific Guidelines**:
- **Astro**: Use SSR capabilities effectively, implement proper API routes
- **React**: Use functional components with hooks, implement proper state management
- **Tailwind CSS v4**: Use utility classes efficiently, follow the project's styling patterns
- **Better Auth**: Implement secure authentication flows, use session validation properly
- **Shadcn UI**: Leverage existing components, maintain consistent UI patterns
- **Drizzle ORM**: Write efficient database queries, use proper schema definitions
5. **Code Review Approach**: When reviewing code:
- Check for adherence to project patterns and CLAUDE.md guidelines
- Identify potential performance issues or bottlenecks
- Suggest improvements for readability and maintainability
- Ensure proper error handling and edge case coverage
- Verify security best practices are followed
6. **Problem-Solving Methodology**:
- Analyze requirements thoroughly before coding
- Break down complex problems into smaller, manageable pieces
- Consider edge cases and error scenarios
- Optimize for both performance and maintainability
- Document complex logic with clear comments
7. **Quality Assurance**:
- Write code that is easy to test
- Consider adding appropriate test cases using Bun's test runner
- Validate inputs and handle errors gracefully
- Ensure code works across different scenarios
**Output Guidelines**:
- Provide complete, working code implementations
- Include clear explanations of design decisions
- Suggest tests when appropriate
- Highlight any potential issues or areas for future improvement
- Follow the existing code style and conventions
**Important Reminders**:
- Never create files unless absolutely necessary
- Always prefer editing existing files
- Don't create documentation unless explicitly requested
- Focus on the specific task at hand
- Reference CLAUDE.md for project-specific patterns and guidelines
You approach every task with the mindset of a seasoned engineer who values code quality, maintainability, and long-term project health. Your solutions should be elegant, efficient, and aligned with the project's established patterns.

View File

@@ -1,61 +0,0 @@
---
name: strategic-task-planner
description: Use this agent when you need to decompose complex projects, features, or problems into structured, actionable plans. This includes breaking down large development tasks, creating implementation roadmaps, organizing multi-step processes, or planning project phases. The agent excels at identifying dependencies, sequencing tasks, and creating clear execution strategies. <example>Context: User needs help planning the implementation of a new feature. user: "I need to add a bulk import feature that can handle CSV files with 100k+ rows" assistant: "I'll use the strategic-task-planner agent to break this down into manageable components and create an implementation plan." <commentary>Since the user is asking about implementing a complex feature, use the Task tool to launch the strategic-task-planner agent to decompose it into actionable steps.</commentary></example> <example>Context: User wants to refactor a large codebase. user: "We need to migrate our entire authentication system from sessions to JWT tokens" assistant: "Let me use the strategic-task-planner agent to create a phased migration plan that minimizes risk." <commentary>Since this is a complex migration requiring careful planning, use the strategic-task-planner agent to create a structured approach.</commentary></example>
tools: Glob, Grep, LS, ExitPlanMode, Read, NotebookRead, WebFetch, TodoWrite, WebSearch, Task, mcp__ide__getDiagnostics, mcp__ide__executeCode, mcp__playwright__browser_close, mcp__playwright__browser_resize, mcp__playwright__browser_console_messages, mcp__playwright__browser_handle_dialog, mcp__playwright__browser_evaluate, mcp__playwright__browser_file_upload, mcp__playwright__browser_install, mcp__playwright__browser_press_key, mcp__playwright__browser_type, mcp__playwright__browser_navigate, mcp__playwright__browser_navigate_back, mcp__playwright__browser_navigate_forward, mcp__playwright__browser_network_requests, mcp__playwright__browser_take_screenshot, mcp__playwright__browser_snapshot, mcp__playwright__browser_click, mcp__playwright__browser_drag, mcp__playwright__browser_hover, mcp__playwright__browser_select_option, mcp__playwright__browser_tab_list, mcp__playwright__browser_tab_new, mcp__playwright__browser_tab_select, mcp__playwright__browser_tab_close, mcp__playwright__browser_wait_for
color: blue
---
You are a strategic planning specialist with deep expertise in decomposing complex tasks and creating actionable execution plans. Your role is to transform ambiguous or overwhelming projects into clear, structured roadmaps that teams can confidently execute.
When analyzing a task or project, you will:
1. **Understand the Core Objective**: Extract the fundamental goal, success criteria, and constraints. Ask clarifying questions if critical details are missing.
2. **Decompose Systematically**: Break down the task using these principles:
- Identify major phases or milestones
- Decompose each phase into concrete, actionable tasks
- Keep tasks small enough to complete in 1-4 hours when possible
- Ensure each task has clear completion criteria
3. **Map Dependencies**: Identify and document:
- Task prerequisites and dependencies
- Critical path items that could block progress
- Parallel work streams that can proceed independently
- Resource or knowledge requirements
4. **Sequence Strategically**: Order tasks by:
- Technical dependencies (what must come first)
- Risk mitigation (tackle unknowns early)
- Value delivery (enable early feedback when possible)
- Resource efficiency (batch similar work)
5. **Provide Actionable Output**: Structure your plans with:
- **Phase Overview**: High-level phases with objectives
- **Task Breakdown**: Numbered tasks with clear descriptions
- **Dependencies**: Explicitly stated prerequisites
- **Effort Estimates**: Rough time estimates when relevant
- **Risk Considerations**: Potential blockers or challenges
- **Success Metrics**: How to measure completion
6. **Adapt to Context**: Tailor your planning approach based on:
- Technical vs non-technical tasks
- Team size and skill level
- Time constraints and deadlines
- Available resources and tools
**Output Format Guidelines**:
- Use clear hierarchical structure (phases → tasks → subtasks)
- Number all tasks for easy reference
- Bold key terms and phase names
- Include time estimates in brackets [2-4 hours]
- Mark critical path items with ⚡
- Flag high-risk items with ⚠️
**Quality Checks**:
- Ensure no task is too large or vague
- Verify all dependencies are identified
- Confirm the plan addresses the original objective
- Check that success criteria are measurable
- Validate that the sequence makes logical sense
Remember: A good plan reduces uncertainty and builds confidence. Focus on clarity, completeness, and actionability. When in doubt, err on the side of breaking things down further rather than leaving ambiguity.

View File

@@ -1,5 +0,0 @@
Evaluate all the updates being made.
Update CHANGELOG.md
Use the chnages in the git log to determine if its a major, minor or a patch release.
Update the package.json first before you push the tag.
Never mention Claude Code in the release notes or in commit messages.

View File

@@ -1,3 +0,0 @@
Generate release notes for the latest release.
Use a temp md file to write the release notes.
Do not check that file into git.

View File

@@ -1,8 +0,0 @@
{
"permissions": {
"allow": [
"Bash(docker build:*)"
],
"deny": []
}
}

View File

@@ -149,7 +149,11 @@ jobs:
### Pull and Test ### Pull and Test
\`\`\`bash \`\`\`bash
docker pull ${imagePath} docker pull ${imagePath}
docker run -d -p 3000:3000 --name gitea-mirror-test ${imagePath} docker run -d \
-p 4321:4321 \
-e BETTER_AUTH_SECRET=your-secret-here \
-e BETTER_AUTH_URL=http://localhost:4321 \
--name gitea-mirror-test ${imagePath}
\`\`\` \`\`\`
### Docker Compose Testing ### Docker Compose Testing
@@ -158,9 +162,11 @@ jobs:
gitea-mirror: gitea-mirror:
image: ${imagePath} image: ${imagePath}
ports: ports:
- "3000:3000" - "4321:4321"
environment: environment:
- BETTER_AUTH_SECRET=your-secret-here - BETTER_AUTH_SECRET=your-secret-here
- BETTER_AUTH_URL=http://localhost:4321
- BETTER_AUTH_TRUSTED_ORIGINS=http://localhost:4321
\`\`\` \`\`\`
> 💡 **Note:** PR images are tagged as \`pr-<number>\` and only built for \`linux/amd64\` to speed up CI. > 💡 **Note:** PR images are tagged as \`pr-<number>\` and only built for \`linux/amd64\` to speed up CI.
@@ -224,4 +230,3 @@ jobs:
continue-on-error: true continue-on-error: true
with: with:
sarif_file: scout-results.sarif sarif_file: scout-results.sarif

459
CLAUDE.md
View File

@@ -2,255 +2,316 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
DONT HALLUCIATE THINGS. IF YOU DONT KNOW LOOK AT THE CODE OR ASK FOR DOCS
NEVER MENTION CLAUDE CODE ANYWHERE.
## Project Overview ## Project Overview
Gitea Mirror is a web application that automatically mirrors repositories from GitHub to self-hosted Gitea instances. It uses Astro for SSR, React for UI, SQLite for data storage, and Bun as the JavaScript runtime. Gitea Mirror is a self-hosted web application that automatically mirrors repositories from GitHub to Gitea instances. It's built with Astro (SSR mode), React, and runs on the Bun runtime with SQLite for data persistence.
## Essential Commands **Key capabilities:**
- Mirrors public, private, and starred GitHub repos to Gitea
- Supports metadata mirroring (issues, PRs as issues, labels, milestones, releases, wiki)
- Git LFS support
- Multiple authentication methods (email/password, OIDC/SSO, header auth)
- Scheduled automatic syncing with configurable intervals
- Auto-discovery of new repos and cleanup of deleted repos
- Multi-user support with encrypted token storage (AES-256-GCM)
## Development Commands
### Setup and Installation
```bash
# Install dependencies
bun install
# Initialize database (first time setup)
bun run setup
# Clean start (reset database)
bun run dev:clean
```
### Development ### Development
```bash ```bash
bun run dev # Start development server (port 3000) # Start development server (http://localhost:4321)
bun run build # Build for production bun run dev
bun run preview # Preview production build
# Build for production
bun run build
# Preview production build
bun run preview
# Start production server
bun run start
``` ```
### Testing ### Testing
```bash ```bash
bun test # Run all tests # Run all tests
bun test:watch # Run tests in watch mode bun test
bun test:coverage # Run tests with coverage
# Run tests in watch mode
bun test:watch
# Run tests with coverage
bun test:coverage
``` ```
**Test configuration:**
- Test runner: Bun's built-in test runner (configured in `bunfig.toml`)
- Setup file: `src/tests/setup.bun.ts` (auto-loaded via bunfig.toml)
- Timeout: 5000ms default
- Tests are colocated with source files using `*.test.ts` pattern
### Database Management ### Database Management
```bash ```bash
bun run init-db # Initialize database # Database operations via Drizzle
bun run reset-users # Reset user accounts (development) bun run db:generate # Generate migrations from schema
bun run cleanup-db # Remove database files bun run db:migrate # Run migrations
bun run db:push # Push schema changes directly
bun run db:studio # Open Drizzle Studio (database GUI)
bun run db:check # Check schema consistency
# Database utilities via custom scripts
bun run manage-db init # Initialize database
bun run manage-db check # Check database health
bun run manage-db fix # Fix database issues
bun run manage-db reset-users # Reset all users
bun run cleanup-db # Delete database file
``` ```
### Production ### Utility Scripts
```bash ```bash
bun run start # Start production server # Recovery and diagnostic scripts
bun run startup-recovery # Recover from crashes
bun run startup-recovery-force # Force recovery
bun run test-recovery # Test recovery mechanism
bun run test-shutdown # Test graceful shutdown
# Environment configuration
bun run startup-env-config # Load config from env vars
``` ```
## Architecture & Key Concepts ## Architecture
### Technology Stack ### Tech Stack
- **Frontend**: Astro (SSR) + React + Tailwind CSS v4 + Shadcn UI - **Frontend:** Astro v5 (SSR mode) + React v19 + Shadcn UI + Tailwind CSS v4
- **Backend**: Bun runtime + SQLite + Drizzle ORM - **Backend:** Astro API routes (Node adapter, standalone mode)
- **APIs**: GitHub (Octokit) and Gitea APIs - **Runtime:** Bun (>=1.2.9)
- **Auth**: Better Auth with email/password, SSO, and OIDC provider support - **Database:** SQLite via Drizzle ORM
- **Authentication:** Better Auth (session-based)
- **APIs:** GitHub (Octokit with throttling plugin), Gitea REST API
### Project Structure ### Directory Structure
- `/src/pages/api/` - API endpoints (Astro API routes)
- `/src/components/` - React components organized by feature ```
- `/src/lib/db/` - Database queries and schema (Drizzle ORM) src/
- `/src/hooks/` - Custom React hooks for data fetching ├── components/ # React components (UI, features)
- `/data/` - SQLite database storage location │ ├── ui/ # Shadcn UI components
│ ├── repositories/ # Repository management components
│ ├── organizations/ # Organization management components
│ └── ...
├── pages/ # Astro pages and API routes
│ ├── api/ # API endpoints (Better Auth integration)
│ │ ├── auth/ # Authentication endpoints
│ │ ├── github/ # GitHub operations
│ │ ├── gitea/ # Gitea operations
│ │ ├── sync/ # Mirror sync operations
│ │ ├── job/ # Job management
│ │ └── ...
│ └── *.astro # Page components
├── lib/ # Core business logic
│ ├── db/ # Database (Drizzle ORM)
│ │ ├── schema.ts # Database schema with Zod validation
│ │ ├── index.ts # Database instance and table exports
│ │ └── adapter.ts # Better Auth SQLite adapter
│ ├── github.ts # GitHub API client (Octokit)
│ ├── gitea.ts # Gitea API client
│ ├── gitea-enhanced.ts # Enhanced Gitea operations (metadata)
│ ├── scheduler-service.ts # Automatic mirroring scheduler
│ ├── cleanup-service.ts # Activity log cleanup
│ ├── repository-cleanup-service.ts # Orphaned repo cleanup
│ ├── auth.ts # Better Auth configuration
│ ├── config.ts # Configuration management
│ ├── helpers.ts # Mirror job creation
│ ├── utils/ # Utility functions
│ │ ├── encryption.ts # AES-256-GCM token encryption
│ │ ├── config-encryption.ts # Config token encryption
│ │ ├── duration-parser.ts # Parse intervals (e.g., "8h", "30m")
│ │ ├── concurrency.ts # Concurrency control utilities
│ │ └── mirror-strategies.ts # Mirror strategy logic
│ └── ...
├── types/ # TypeScript type definitions
├── tests/ # Test utilities and setup
└── middleware.ts # Astro middleware (auth, session)
scripts/ # Utility scripts
├── manage-db.ts # Database management CLI
├── startup-recovery.ts # Crash recovery
└── ...
```
### Key Architectural Patterns ### Key Architectural Patterns
1. **API Routes**: All API endpoints follow the pattern `/api/[resource]/[action]` and use `createSecureErrorResponse` for consistent error handling: #### 1. Database Schema and Validation
```typescript - **Location:** `src/lib/db/schema.ts`
import { createSecureErrorResponse } from '@/lib/utils/error-handler'; - **Pattern:** Drizzle ORM tables + Zod schemas for validation
- **Key tables:**
- `configs` - User configuration (GitHub/Gitea settings, mirror options)
- `repositories` - Tracked repositories with metadata
- `organizations` - GitHub organizations with destination overrides
- `mirrorJobs` - Mirror job queue and history
- `activities` - Activity log for dashboard
- `user`, `session`, `account` - Better Auth tables
export async function POST({ request }: APIContext) { **Important:** All config tokens (GitHub/Gitea) are encrypted at rest using AES-256-GCM. Use helper functions from `src/lib/utils/config-encryption.ts` to decrypt.
try {
// Implementation
} catch (error) {
return createSecureErrorResponse(error);
}
}
```
2. **Database Queries**: Located in `/src/lib/db/queries/` organized by domain (users, repositories, etc.) #### 2. Mirror Job System
- **Location:** `src/lib/helpers.ts` (createMirrorJob)
- **Flow:**
1. User triggers mirror via API endpoint
2. `createMirrorJob()` creates job record with status "pending"
3. Job processor (in API routes) performs GitHub → Gitea operations
4. Job status updated throughout: "mirroring" → "success"/"failed"
5. Events published via SSE for real-time UI updates
3. **Real-time Updates**: Server-Sent Events (SSE) endpoint at `/api/events` for live dashboard updates #### 3. GitHub ↔ Gitea Mirroring
- **GitHub Client:** `src/lib/github.ts` - Octokit with rate limit tracking
- **Gitea Client:** `src/lib/gitea.ts` - Basic repo operations
- **Enhanced Gitea:** `src/lib/gitea-enhanced.ts` - Metadata mirroring (issues, PRs, releases)
4. **Authentication System**: **Mirror strategies (configured per user):**
- Built on Better Auth library - `preserve` - Maintain GitHub org structure in Gitea
- Three authentication methods: - `single-org` - All repos into one Gitea org
- Email & Password (traditional auth) - `flat-user` - All repos under user account
- SSO (authenticate via external OIDC providers) - `mixed` - Personal repos in one org, org repos preserve structure
- OIDC Provider (act as OIDC provider for other apps)
- Session-based authentication with secure cookies
- First user signup creates admin account
- Protected routes use Better Auth session validation
5. **Mirror Process**: **Metadata mirroring:**
- Discovers repos from GitHub (user/org) - Issues transferred with comments, labels, assignees
- Creates/updates mirror in Gitea - PRs converted to issues (Gitea API limitation - cannot create PRs)
- Tracks status in database - Tagged with "pull-request" label
- Supports scheduled automatic mirroring - Title prefixed with `[PR #number] [STATUS]`
- Body includes commit history, file changes, merge status
- Releases mirrored with assets
- Labels and milestones preserved
- Wiki content cloned if enabled
- **Sequential processing:** Issues/PRs mirrored one at a time to prevent out-of-order creation (see `src/lib/gitea-enhanced.ts`)
6. **Mirror Strategies**: Four ways to organize repositories in Gitea: #### 4. Scheduler Service
- **preserve**: Maintains GitHub structure (default) - **Location:** `src/lib/scheduler-service.ts`
- Organization repos → Same organization name in Gitea - **Features:**
- Personal repos → Under your Gitea username - Cron-based or interval-based scheduling (uses `duration-parser.ts`)
- **single-org**: All repos go to one organization - Auto-start on boot when `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` is set
- All repos → Single configured organization - Auto-import new GitHub repos
- **flat-user**: All repos go under user account - Auto-cleanup orphaned repos (archive or delete)
- All repos → Under your Gitea username - Respects per-repo mirror intervals (not Gitea's default 24h)
- **mixed**: Hybrid approach - **Concurrency control:** Uses `src/lib/utils/concurrency.ts` for batch processing
- Organization repos → Preserve structure
- Personal repos → Single configured organization
- Starred repos always go to separate organization (starredReposOrg, default: "starred")
- Routing logic in `getGiteaRepoOwner()` function
### Database Schema (SQLite) #### 5. Authentication System
- `users` - User accounts and authentication - **Location:** `src/lib/auth.ts`, `src/lib/auth-client.ts`
- `configs` - GitHub/Gitea connection settings - **Better Auth integration:**
- `repositories` - Repository mirror status and metadata - Email/password (always enabled)
- `organizations` - Organization structure preservation - OIDC/SSO providers (configurable via UI)
- `mirror_jobs` - Scheduled mirror operations - Header authentication for reverse proxies (Authentik, Authelia)
- `events` - Activity log and notifications - **Session management:** Cookie-based, validated in Astro middleware
- **User helpers:** `src/lib/utils/auth-helpers.ts`
### Testing Approach #### 6. Environment Configuration
- Uses Bun's native test runner (`bun:test`) - **Startup:** `src/lib/env-config-loader.ts` + `scripts/startup-env-config.ts`
- Test files use `.test.ts` or `.test.tsx` extension - **Pattern:** Environment variables can pre-configure settings, but users can override via web UI
- Setup file at `/src/tests/setup.bun.ts` - **Encryption:** `ENCRYPTION_SECRET` for tokens, `BETTER_AUTH_SECRET` for sessions
- Mock utilities available for API testing.
### Development Tips #### 7. Real-time Updates
- Environment variables in `.env` (copy from `.env.example`) - **Events:** `src/lib/events.ts` + `src/lib/events/realtime.ts`
- BETTER_AUTH_SECRET required for session signing - **Pattern:** Server-Sent Events (SSE) for live dashboard updates
- Database auto-initializes on first run - **Endpoints:** `/api/sse` - client subscribes to job/repo events
- Use `bun run dev:clean` for fresh database start
- Tailwind CSS v4 configured with Vite plugin
### Authentication Setup ### Testing Patterns
- **Better Auth** handles all authentication
- Configuration in `/src/lib/auth.ts` (server) and `/src/lib/auth-client.ts` (client)
- Auth endpoints available at `/api/auth/*`
- SSO providers configured through the web UI
- OIDC provider functionality for external applications
### Common Tasks **Unit tests:**
- Colocated with source: `filename.test.ts` alongside `filename.ts`
- Use Bun's built-in assertions and mocking
- Mock external APIs (GitHub, Gitea) using `src/tests/mock-fetch.ts`
**Adding a new API endpoint:** **Integration tests:**
1. Create file in `/src/pages/api/[resource]/[action].ts` - Located in `src/tests/`
2. Use `createSecureErrorResponse` for error handling - Test database operations with in-memory SQLite
3. Add corresponding database query in `/src/lib/db/queries/` - Example: `src/lib/db/index.test.ts`
4. Update types in `/src/types/` if needed
**Adding a new component:** **Test utilities:**
1. Create in appropriate `/src/components/[feature]/` directory - `src/tests/setup.bun.ts` - Global test setup (loaded via bunfig.toml)
2. Use Shadcn UI components from `/src/components/ui/` - `src/tests/mock-fetch.ts` - Fetch mocking utilities
3. Follow existing naming patterns (e.g., `RepositoryCard`, `ConfigTabs`)
**Modifying database schema:** ### Important Development Notes
1. Update schema in `/src/lib/db/schema.ts`
2. Run `bun run init-db` to recreate database
3. Update related queries in `/src/lib/db/queries/`
## Configuration Options 1. **Path Aliases:** Use `@/` for imports (configured in `tsconfig.json`)
```typescript
import { db } from '@/lib/db';
```
### GitHub Configuration (UI Fields) 2. **Token Encryption:** Always use encryption helpers when dealing with tokens:
```typescript
import { getDecryptedGitHubToken, getDecryptedGiteaToken } from '@/lib/utils/config-encryption';
```
#### Basic Settings (`githubConfig`) 3. **API Route Pattern:** Astro API routes in `src/pages/api/` should:
- **username**: GitHub username - Check authentication via Better Auth
- **token**: GitHub personal access token (requires repo and admin:org scopes) - Validate input with Zod schemas
- **privateRepositories**: Include private repositories - Handle errors gracefully
- **mirrorStarred**: Mirror starred repositories - Return JSON responses
### Gitea Configuration (UI Fields) 4. **Database Migrations:**
- **url**: Gitea instance URL - Schema changes: Update `src/lib/db/schema.ts`
- **username**: Gitea username - Generate migration: `bun run db:generate`
- **token**: Gitea access token - Review generated SQL in `drizzle/` directory
- **organization**: Destination organization (for single-org/mixed strategies) - Apply: `bun run db:migrate` (or `db:push` for dev)
- **starredReposOrg**: Organization for starred repositories (default: "starred")
- **visibility**: Organization visibility - "public", "private", "limited"
- **mirrorStrategy**: Repository organization strategy (set via UI)
- **preserveOrgStructure**: Automatically set based on mirrorStrategy
### Schedule Configuration (`scheduleConfig`) 5. **Concurrency Control:**
- **enabled**: Enable automatic mirroring (default: false) - Use utilities from `src/lib/utils/concurrency.ts` for batch operations
- **interval**: Cron expression or seconds (default: "0 2 * * *" - 2 AM daily) - Respect rate limits (GitHub: 5000 req/hr authenticated, Gitea: varies)
- **concurrent**: Allow concurrent mirror operations (default: false) - Issue/PR mirroring is sequential to maintain chronological order
- **batchSize**: Number of repos to process in parallel (default: 10)
### Database Cleanup Configuration (`cleanupConfig`) 6. **Duration Parsing:**
- **enabled**: Enable automatic cleanup (default: false) - Use `parseInterval()` from `src/lib/utils/duration-parser.ts`
- **retentionDays**: Days to keep events (stored as seconds internally) - Supports: "30m", "8h", "24h", "7d", cron expressions, or milliseconds
### Mirror Options (UI Fields) 7. **Graceful Shutdown:**
- **mirrorReleases**: Mirror GitHub releases to Gitea - Services implement cleanup handlers (see `src/lib/shutdown-manager.ts`)
- **mirrorLFS**: Mirror Git LFS (Large File Storage) objects - Recovery system in `src/lib/recovery.ts` handles interrupted jobs
- Requires LFS enabled on Gitea server (LFS_START_SERVER = true)
- Requires Git v2.1.2+ on server
- **mirrorMetadata**: Enable metadata mirroring (master toggle)
- **metadataComponents** (only available when mirrorMetadata is enabled):
- **issues**: Mirror issues
- **pullRequests**: Mirror pull requests
- **labels**: Mirror labels
- **milestones**: Mirror milestones
- **wiki**: Mirror wiki content
### Advanced Options (UI Fields) ## Common Development Workflows
- **skipForks**: Skip forked repositories (default: false)
- **starredCodeOnly**: Skip issues for starred repositories (default: false) - enables "Lightweight mode" for starred repos
### Repository Statuses ### Adding a new mirror option
Repositories can have the following statuses: 1. Update Zod schema in `src/lib/db/schema.ts` (e.g., `giteaConfigSchema`)
- **imported**: Repository discovered from GitHub 2. Update TypeScript types in `src/types/config.ts`
- **mirroring**: Currently being mirrored to Gitea 3. Add UI control in settings page component
- **mirrored**: Successfully mirrored 4. Update API handler in `src/pages/api/config/`
- **syncing**: Repository being synchronized 5. Implement logic in `src/lib/gitea.ts` or `src/lib/gitea-enhanced.ts`
- **synced**: Successfully synchronized
- **failed**: Mirror/sync operation failed
- **skipped**: Skipped due to filters or conditions
- **ignored**: User explicitly marked to ignore (won't be mirrored/synced)
- **deleting**: Repository being deleted
- **deleted**: Repository deleted
### Scheduling and Synchronization (Issue #72 Fixes) ### Debugging mirror failures
1. Check mirror jobs: `bun run db:studio` → `mirrorJobs` table
2. Review activity logs: Dashboard → Activity tab
3. Check console logs for API errors (GitHub/Gitea rate limits, auth issues)
4. Use diagnostic scripts: `bun run test-recovery`
#### Fixed Issues ### Adding authentication provider
1. **Mirror Interval Bug**: Added `mirror_interval` parameter to Gitea API calls when creating mirrors (previously defaulted to 24h) 1. Update Better Auth config in `src/lib/auth.ts`
2. **Auto-Discovery**: Scheduler now automatically discovers and imports new GitHub repositories 2. Add provider configuration UI in settings
3. **Interval Updates**: Sync operations now update existing mirrors' intervals to match configuration 3. Test with `src/tests/test-gitea-auth.ts` patterns
4. **Repository Cleanup**: Integrated automatic cleanup of orphaned repositories (repos removed from GitHub) 4. Update documentation in `docs/SSO-OIDC-SETUP.md`
#### Environment Variables for Auto-Import ## Docker Deployment
- **AUTO_IMPORT_REPOS**: Set to `false` to disable automatic repository discovery (default: enabled)
#### How Scheduling Works - **Dockerfile:** Multi-stage build (bun base → build → production)
- **Scheduler Service**: Runs every minute to check for scheduled tasks - **Entrypoint:** `docker-entrypoint.sh` - handles CA certs, user permissions, database init
- **Sync Interval**: Configured via `GITEA_MIRROR_INTERVAL` or UI (e.g., "8h", "30m", "1d") - **Compose files:**
- **Auto-Import**: Checks GitHub for new repositories during each scheduled sync - `docker-compose.alt.yml` - Quick start (pre-built image, minimal config)
- **Auto-Cleanup**: Removes repositories that no longer exist in GitHub (if enabled) - `docker-compose.yml` - Full setup (build from source, all env vars)
- **Mirror Interval Update**: Updates Gitea's internal mirror interval during sync operations - `docker-compose.dev.yml` - Development with hot reload
### Authentication Configuration ## Additional Resources
#### SSO Provider Configuration - **Environment Variables:** See `docs/ENVIRONMENT_VARIABLES.md` for complete list
- **issuerUrl**: OIDC issuer URL (e.g., https://accounts.google.com) - **Development Workflow:** See `docs/DEVELOPMENT_WORKFLOW.md`
- **domain**: Email domain for this provider - **SSO Setup:** See `docs/SSO-OIDC-SETUP.md`
- **providerId**: Unique identifier for the provider - **Contributing:** See `CONTRIBUTING.md` for code guidelines and scope
- **clientId**: OAuth client ID from provider - **Graceful Shutdown:** See `docs/GRACEFUL_SHUTDOWN.md` for crash recovery details
- **clientSecret**: OAuth client secret from provider
- **authorizationEndpoint**: OAuth authorization URL (auto-discovered if supported)
- **tokenEndpoint**: OAuth token exchange URL (auto-discovered if supported)
- **jwksEndpoint**: JSON Web Key Set URL (optional, auto-discovered)
- **userInfoEndpoint**: User information endpoint (optional, auto-discovered)
#### OIDC Provider Settings (for external apps)
- **allowedRedirectUris**: Comma-separated list of allowed redirect URIs
- **clientId**: Generated client ID for the application
- **clientSecret**: Generated client secret for the application
- **scopes**: Available scopes (openid, profile, email)
#### Environment Variables
- **BETTER_AUTH_SECRET**: Secret key for signing sessions (required)
- **BETTER_AUTH_URL**: Base URL for authentication (default: http://localhost:4321)
## Security Guidelines
- **Confidentiality Guidelines**:
- Dont ever say Claude Code or generated with AI anyhwere.
- Never commit without the explicict ask

View File

@@ -10,10 +10,6 @@
</p> </p>
</p> </p>
> [!IMPORTANT]
> **Upgrading to v3?** v3 requires a fresh start with a new data volume. Please read the [Upgrade Guide](UPGRADE.md) for instructions.
## 🚀 Quick Start ## 🚀 Quick Start
```bash ```bash

View File

@@ -1,74 +0,0 @@
# Upgrade Guide
## Upgrading to v3.0
> **⚠️ IMPORTANT**: v3.0 requires a fresh start. There is no automated migration from v2.x to v3.0.
### Why No Migration?
v3.0 introduces fundamental changes to the application architecture:
- **Authentication**: Switched from JWT to Better Auth
- **Database**: Now uses Drizzle ORM with proper migrations
- **Security**: All tokens are now encrypted
- **Features**: Added SSO support and OIDC provider functionality
Due to these extensive changes, we recommend starting fresh with v3.0 for the best experience.
### Upgrade Steps
1. **Stop your v2.x container**
```bash
docker stop gitea-mirror
docker rm gitea-mirror
```
2. **Backup your v2.x data (optional)**
```bash
# If you want to keep your v2 data for reference
docker run --rm -v gitea-mirror-data:/data -v $(pwd):/backup alpine tar czf /backup/gitea-mirror-v2-backup.tar.gz -C /data .
```
3. **Create a new volume for v3**
```bash
docker volume create gitea-mirror-v3-data
```
4. **Run v3 with the new volume**
```bash
docker run -d \
--name gitea-mirror \
-p 4321:4321 \
-v gitea-mirror-v3-data:/app/data \
-e BETTER_AUTH_SECRET=your-secret-key \
-e ENCRYPTION_SECRET=your-encryption-key \
arunavo4/gitea-mirror:latest
```
5. **Set up your configuration again**
- Navigate to http://localhost:4321
- Create a new admin account
- Re-enter your GitHub and Gitea credentials
- Configure your mirror settings
### What Happens to My Existing Mirrors?
Your existing mirrors in Gitea are **not affected**. The application will:
- Recognize existing repositories when you re-import
- Skip creating duplicates
- Resume normal mirror operations
### Environment Variable Changes
v3.0 uses different environment variables:
| v2.x | v3.0 | Notes |
|------|------|-------|
| `JWT_SECRET` | `BETTER_AUTH_SECRET` | Required for session management |
| - | `ENCRYPTION_SECRET` | New - required for token encryption |
### Need Help?
If you have questions about upgrading:
1. Check the [README](README.md) for v3 setup instructions
2. Review your v2 configuration before upgrading
3. Open an issue if you encounter problems

View File

@@ -26,6 +26,10 @@ services:
- HOST=0.0.0.0 - HOST=0.0.0.0
- PORT=4321 - PORT=4321
- PUBLIC_BETTER_AUTH_URL=${PUBLIC_BETTER_AUTH_URL:-http://localhost:4321} - PUBLIC_BETTER_AUTH_URL=${PUBLIC_BETTER_AUTH_URL:-http://localhost:4321}
# Optional concurrency controls (defaults match in-app defaults)
# If you want perfect ordering of issues and PRs, set these at 1
- MIRROR_ISSUE_CONCURRENCY=${MIRROR_ISSUE_CONCURRENCY:-3}
- MIRROR_PULL_REQUEST_CONCURRENCY=${MIRROR_PULL_REQUEST_CONCURRENCY:-5}
healthcheck: healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=3", "--spider", "http://localhost:4321/api/health"] test: ["CMD", "wget", "--no-verbose", "--tries=3", "--spider", "http://localhost:4321/api/health"]
@@ -54,4 +58,4 @@ services:
# - Auto-import settings # - Auto-import settings
# - Cleanup preferences # - Cleanup preferences
# #
# That's it! Everything else can be configured via the web interface. # That's it! Everything else can be configured via the web interface.

View File

@@ -47,6 +47,8 @@ services:
- PRESERVE_ORG_STRUCTURE=${PRESERVE_ORG_STRUCTURE:-false} - PRESERVE_ORG_STRUCTURE=${PRESERVE_ORG_STRUCTURE:-false}
- ONLY_MIRROR_ORGS=${ONLY_MIRROR_ORGS:-false} - ONLY_MIRROR_ORGS=${ONLY_MIRROR_ORGS:-false}
- SKIP_STARRED_ISSUES=${SKIP_STARRED_ISSUES:-false} - SKIP_STARRED_ISSUES=${SKIP_STARRED_ISSUES:-false}
- MIRROR_ISSUE_CONCURRENCY=${MIRROR_ISSUE_CONCURRENCY:-3}
- MIRROR_PULL_REQUEST_CONCURRENCY=${MIRROR_PULL_REQUEST_CONCURRENCY:-5}
- GITEA_URL=${GITEA_URL:-} - GITEA_URL=${GITEA_URL:-}
- GITEA_TOKEN=${GITEA_TOKEN:-} - GITEA_TOKEN=${GITEA_TOKEN:-}
- GITEA_USERNAME=${GITEA_USERNAME:-} - GITEA_USERNAME=${GITEA_USERNAME:-}

View File

@@ -141,6 +141,10 @@ Control what content gets mirrored from GitHub to Gitea.
| `MIRROR_PULL_REQUESTS` | Mirror pull requests (requires MIRROR_METADATA=true) | `false` | `true`, `false` | | `MIRROR_PULL_REQUESTS` | Mirror pull requests (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_LABELS` | Mirror labels (requires MIRROR_METADATA=true) | `false` | `true`, `false` | | `MIRROR_LABELS` | Mirror labels (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_MILESTONES` | Mirror milestones (requires MIRROR_METADATA=true) | `false` | `true`, `false` | | `MIRROR_MILESTONES` | Mirror milestones (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
| `MIRROR_ISSUE_CONCURRENCY` | Number of issues processed in parallel. Set above `1` to speed up mirroring at the risk of out-of-order creation. | `3` | Integer ≥ 1 |
| `MIRROR_PULL_REQUEST_CONCURRENCY` | Number of pull requests processed in parallel. Values above `1` may cause ordering differences. | `5` | Integer ≥ 1 |
> **Ordering vs Throughput:** Metadata now mirrors sequentially by default to preserve chronology. Increase the concurrency variables only if you can tolerate minor out-of-order entries.
## Automation Configuration ## Automation Configuration

View File

@@ -54,6 +54,8 @@ export const giteaConfigSchema = z.object({
.enum(["skip", "reference", "full-copy"]) .enum(["skip", "reference", "full-copy"])
.default("reference"), .default("reference"),
// Mirror options // Mirror options
issueConcurrency: z.number().int().min(1).default(3),
pullRequestConcurrency: z.number().int().min(1).default(5),
mirrorReleases: z.boolean().default(false), mirrorReleases: z.boolean().default(false),
releaseLimit: z.number().default(10), releaseLimit: z.number().default(10),
mirrorMetadata: z.boolean().default(false), mirrorMetadata: z.boolean().default(false),

View File

@@ -49,6 +49,9 @@ interface EnvConfig {
mirrorLabels?: boolean; mirrorLabels?: boolean;
mirrorMilestones?: boolean; mirrorMilestones?: boolean;
mirrorMetadata?: boolean; mirrorMetadata?: boolean;
releaseLimit?: number;
issueConcurrency?: number;
pullRequestConcurrency?: number;
}; };
schedule: { schedule: {
enabled?: boolean; enabled?: boolean;
@@ -136,6 +139,8 @@ function parseEnvConfig(): EnvConfig {
mirrorMilestones: process.env.MIRROR_MILESTONES === 'true', mirrorMilestones: process.env.MIRROR_MILESTONES === 'true',
mirrorMetadata: process.env.MIRROR_METADATA === 'true', mirrorMetadata: process.env.MIRROR_METADATA === 'true',
releaseLimit: process.env.RELEASE_LIMIT ? parseInt(process.env.RELEASE_LIMIT, 10) : undefined, releaseLimit: process.env.RELEASE_LIMIT ? parseInt(process.env.RELEASE_LIMIT, 10) : undefined,
issueConcurrency: process.env.MIRROR_ISSUE_CONCURRENCY ? parseInt(process.env.MIRROR_ISSUE_CONCURRENCY, 10) : undefined,
pullRequestConcurrency: process.env.MIRROR_PULL_REQUEST_CONCURRENCY ? parseInt(process.env.MIRROR_PULL_REQUEST_CONCURRENCY, 10) : undefined,
}, },
schedule: { schedule: {
enabled: process.env.SCHEDULE_ENABLED === 'true' || enabled: process.env.SCHEDULE_ENABLED === 'true' ||
@@ -277,6 +282,12 @@ export async function initializeConfigFromEnv(): Promise<void> {
// Mirror metadata options // Mirror metadata options
mirrorReleases: envConfig.mirror.mirrorReleases ?? existingConfig?.[0]?.giteaConfig?.mirrorReleases ?? false, mirrorReleases: envConfig.mirror.mirrorReleases ?? existingConfig?.[0]?.giteaConfig?.mirrorReleases ?? false,
releaseLimit: envConfig.mirror.releaseLimit ?? existingConfig?.[0]?.giteaConfig?.releaseLimit ?? 10, releaseLimit: envConfig.mirror.releaseLimit ?? existingConfig?.[0]?.giteaConfig?.releaseLimit ?? 10,
issueConcurrency: envConfig.mirror.issueConcurrency && envConfig.mirror.issueConcurrency > 0
? envConfig.mirror.issueConcurrency
: existingConfig?.[0]?.giteaConfig?.issueConcurrency ?? 3,
pullRequestConcurrency: envConfig.mirror.pullRequestConcurrency && envConfig.mirror.pullRequestConcurrency > 0
? envConfig.mirror.pullRequestConcurrency
: existingConfig?.[0]?.giteaConfig?.pullRequestConcurrency ?? 5,
mirrorMetadata: envConfig.mirror.mirrorMetadata ?? (envConfig.mirror.mirrorIssues || envConfig.mirror.mirrorPullRequests || envConfig.mirror.mirrorLabels || envConfig.mirror.mirrorMilestones) ?? existingConfig?.[0]?.giteaConfig?.mirrorMetadata ?? false, mirrorMetadata: envConfig.mirror.mirrorMetadata ?? (envConfig.mirror.mirrorIssues || envConfig.mirror.mirrorPullRequests || envConfig.mirror.mirrorLabels || envConfig.mirror.mirrorMilestones) ?? existingConfig?.[0]?.giteaConfig?.mirrorMetadata ?? false,
mirrorIssues: envConfig.mirror.mirrorIssues ?? existingConfig?.[0]?.giteaConfig?.mirrorIssues ?? false, mirrorIssues: envConfig.mirror.mirrorIssues ?? existingConfig?.[0]?.giteaConfig?.mirrorIssues ?? false,
mirrorPullRequests: envConfig.mirror.mirrorPullRequests ?? existingConfig?.[0]?.giteaConfig?.mirrorPullRequests ?? false, mirrorPullRequests: envConfig.mirror.mirrorPullRequests ?? existingConfig?.[0]?.giteaConfig?.mirrorPullRequests ?? false,

View File

@@ -12,6 +12,7 @@ import { createMirrorJob } from "./helpers";
import { db, organizations, repositories } from "./db"; import { db, organizations, repositories } from "./db";
import { eq, and } from "drizzle-orm"; import { eq, and } from "drizzle-orm";
import { decryptConfigTokens } from "./utils/config-encryption"; import { decryptConfigTokens } from "./utils/config-encryption";
import { formatDateShort } from "./utils";
/** /**
* Helper function to get organization configuration including destination override * Helper function to get organization configuration including destination override
@@ -1558,6 +1559,8 @@ export const mirrorGitRepoIssuesToGitea = async ({
repo, repo,
state: "all", state: "all",
per_page: 100, per_page: 100,
sort: "created",
direction: "asc",
}, },
(res) => res.data (res) => res.data
); );
@@ -1590,6 +1593,18 @@ export const mirrorGitRepoIssuesToGitea = async ({
// Import the processWithRetry function // Import the processWithRetry function
const { processWithRetry } = await import("@/lib/utils/concurrency"); const { processWithRetry } = await import("@/lib/utils/concurrency");
const rawIssueConcurrency = config.giteaConfig?.issueConcurrency ?? 3;
const issueConcurrencyLimit =
Number.isFinite(rawIssueConcurrency)
? Math.max(1, Math.floor(rawIssueConcurrency))
: 1;
if (issueConcurrencyLimit > 1) {
console.warn(
`[Issues] Concurrency is set to ${issueConcurrencyLimit}. This may lead to out-of-order issue creation in Gitea but is faster.`
);
}
// Process issues in parallel with concurrency control // Process issues in parallel with concurrency control
await processWithRetry( await processWithRetry(
filteredIssues, filteredIssues,
@@ -1632,11 +1647,15 @@ export const mirrorGitRepoIssuesToGitea = async ({
.join(", ")} on GitHub.` .join(", ")} on GitHub.`
: ""; : "";
const issueAuthor = issue.user?.login ?? "unknown";
const issueCreatedOn = formatDateShort(issue.created_at);
const issueOriginHeader = `Originally created by @${issueAuthor} on GitHub${
issueCreatedOn ? ` (${issueCreatedOn})` : ""
}.`;
const issuePayload: any = { const issuePayload: any = {
title: issue.title, title: issue.title,
body: `Originally created by @${ body: `${issueOriginHeader}${originalAssignees}\n\n${issue.body ?? ""}`,
issue.user?.login
} on GitHub.${originalAssignees}\n\n${issue.body || ""}`,
closed: issue.state === "closed", closed: issue.state === "closed",
labels: giteaLabelIds, labels: giteaLabelIds,
}; };
@@ -1662,15 +1681,30 @@ export const mirrorGitRepoIssuesToGitea = async ({
(res) => res.data (res) => res.data
); );
// Process comments in parallel with concurrency control // Ensure comments are applied in chronological order to preserve discussion flow
if (comments.length > 0) { const sortedComments = comments
.slice()
.sort(
(a, b) =>
new Date(a.created_at || 0).getTime() -
new Date(b.created_at || 0).getTime()
);
// Process comments sequentially to preserve historical ordering
if (sortedComments.length > 0) {
await processWithRetry( await processWithRetry(
comments, sortedComments,
async (comment) => { async (comment) => {
const commenter = comment.user?.login ?? "unknown";
const commentDate = formatDateShort(comment.created_at);
const commentHeader = `@${commenter} commented on GitHub${
commentDate ? ` (${commentDate})` : ""
}:`;
await httpPost( await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${createdIssue.data.number}/comments`, `${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${createdIssue.data.number}/comments`,
{ {
body: `@${comment.user?.login} commented on GitHub:\n\n${comment.body}`, body: `${commentHeader}\n\n${comment.body ?? ""}`,
}, },
{ {
Authorization: `token ${decryptedConfig.giteaConfig!.token}`, Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
@@ -1679,7 +1713,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
return comment; return comment;
}, },
{ {
concurrencyLimit: 5, concurrencyLimit: 1,
maxRetries: 2, maxRetries: 2,
retryDelay: 1000, retryDelay: 1000,
onRetry: (_comment, error, attempt) => { onRetry: (_comment, error, attempt) => {
@@ -1694,7 +1728,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
return issue; return issue;
}, },
{ {
concurrencyLimit: 3, // Process 3 issues at a time concurrencyLimit: issueConcurrencyLimit,
maxRetries: 2, maxRetries: 2,
retryDelay: 2000, retryDelay: 2000,
onProgress: (completed, total, result) => { onProgress: (completed, total, result) => {
@@ -1966,6 +2000,8 @@ export async function mirrorGitRepoPullRequestsToGitea({
repo, repo,
state: "all", state: "all",
per_page: 100, per_page: 100,
sort: "created",
direction: "asc",
}, },
(res) => res.data (res) => res.data
); );
@@ -2022,6 +2058,18 @@ export async function mirrorGitRepoPullRequestsToGitea({
const { processWithRetry } = await import("@/lib/utils/concurrency"); const { processWithRetry } = await import("@/lib/utils/concurrency");
const rawPullConcurrency = config.giteaConfig?.pullRequestConcurrency ?? 5;
const pullRequestConcurrencyLimit =
Number.isFinite(rawPullConcurrency)
? Math.max(1, Math.floor(rawPullConcurrency))
: 1;
if (pullRequestConcurrencyLimit > 1) {
console.warn(
`[Pull Requests] Concurrency is set to ${pullRequestConcurrencyLimit}. This may lead to out-of-order pull request mirroring in Gitea.`
);
}
let successCount = 0; let successCount = 0;
let failedCount = 0; let failedCount = 0;
@@ -2144,7 +2192,7 @@ export async function mirrorGitRepoPullRequestsToGitea({
} }
}, },
{ {
concurrencyLimit: 5, concurrencyLimit: pullRequestConcurrencyLimit,
maxRetries: 3, maxRetries: 3,
retryDelay: 1000, retryDelay: 1000,
} }

View File

@@ -1,5 +1,5 @@
import { describe, test, expect } from "bun:test"; import { describe, test, expect } from "bun:test";
import { jsonResponse, formatDate, truncate, safeParse, parseErrorMessage, showErrorToast } from "./utils"; import { jsonResponse, formatDate, formatDateShort, truncate, safeParse, parseErrorMessage, showErrorToast } from "./utils";
describe("jsonResponse", () => { describe("jsonResponse", () => {
test("creates a Response with JSON content", () => { test("creates a Response with JSON content", () => {
@@ -65,6 +65,18 @@ describe("formatDate", () => {
}); });
}); });
describe("formatDateShort", () => {
test("returns formatted date when input is provided", () => {
const formatted = formatDateShort("2014-10-20T15:32:10Z");
expect(formatted).toBe("Oct 20, 2014");
});
test("returns undefined when date is missing", () => {
expect(formatDateShort(null)).toBeUndefined();
expect(formatDateShort(undefined)).toBeUndefined();
});
});
describe("truncate", () => { describe("truncate", () => {
test("truncates a string that exceeds the length", () => { test("truncates a string that exceeds the length", () => {
const str = "This is a long string that needs truncation"; const str = "This is a long string that needs truncation";

View File

@@ -29,6 +29,15 @@ export function formatDate(date?: Date | string | null): string {
}).format(new Date(date)); }).format(new Date(date));
} }
export function formatDateShort(date?: Date | string | null): string | undefined {
if (!date) return undefined;
return new Intl.DateTimeFormat("en-US", {
year: "numeric",
month: "short",
day: "numeric",
}).format(new Date(date));
}
export function formatLastSyncTime(date: Date | string | null): string { export function formatLastSyncTime(date: Date | string | null): string {
if (!date) return "Never"; if (!date) return "Never";

View File

@@ -86,6 +86,8 @@ export async function createDefaultConfig({ userId, envOverrides = {} }: Default
addTopics: true, addTopics: true,
preserveVisibility: false, preserveVisibility: false,
forkStrategy: "reference", forkStrategy: "reference",
issueConcurrency: 3,
pullRequestConcurrency: 5,
}, },
include: [], include: [],
exclude: [], exclude: [],

View File

@@ -89,6 +89,8 @@ export function mapUiToDbConfig(
forkStrategy: advancedOptions.skipForks ? "skip" : "reference", forkStrategy: advancedOptions.skipForks ? "skip" : "reference",
// Mirror options from UI // Mirror options from UI
issueConcurrency: giteaConfig.issueConcurrency ?? 3,
pullRequestConcurrency: giteaConfig.pullRequestConcurrency ?? 5,
mirrorReleases: mirrorOptions.mirrorReleases, mirrorReleases: mirrorOptions.mirrorReleases,
releaseLimit: mirrorOptions.releaseLimit || 10, releaseLimit: mirrorOptions.releaseLimit || 10,
mirrorMetadata: mirrorOptions.mirrorMetadata, mirrorMetadata: mirrorOptions.mirrorMetadata,
@@ -132,6 +134,8 @@ export function mapDbToUiConfig(dbConfig: any): {
preserveOrgStructure: dbConfig.giteaConfig?.preserveVisibility || false, // Map preserveVisibility preserveOrgStructure: dbConfig.giteaConfig?.preserveVisibility || false, // Map preserveVisibility
mirrorStrategy: dbConfig.githubConfig?.mirrorStrategy || "preserve", // Get from GitHub config mirrorStrategy: dbConfig.githubConfig?.mirrorStrategy || "preserve", // Get from GitHub config
personalReposOrg: undefined, // Not stored in current schema personalReposOrg: undefined, // Not stored in current schema
issueConcurrency: dbConfig.giteaConfig?.issueConcurrency ?? 3,
pullRequestConcurrency: dbConfig.giteaConfig?.pullRequestConcurrency ?? 5,
}; };
// Map mirror options from various database fields // Map mirror options from various database fields

View File

@@ -13,6 +13,8 @@ export interface GiteaConfig {
preserveOrgStructure: boolean; preserveOrgStructure: boolean;
mirrorStrategy?: MirrorStrategy; // New field for the strategy mirrorStrategy?: MirrorStrategy; // New field for the strategy
personalReposOrg?: string; // Override destination for personal repos personalReposOrg?: string; // Override destination for personal repos
issueConcurrency?: number;
pullRequestConcurrency?: number;
} }
export interface ScheduleConfig { export interface ScheduleConfig {