mirror of
https://github.com/RayLabsHQ/gitea-mirror.git
synced 2025-12-06 03:26:44 +03:00
Merge pull request #128 from RayLabsHQ/fix/chronological-metadata-ordering
fix: preserve chronological issue mirroring
This commit is contained in:
ARUNAVO RAY
@@ -1,76 +0,0 @@
|
||||
---
|
||||
name: qa-testing-specialist
|
||||
description: Use this agent when you need to review code for testability, create comprehensive test strategies, write test cases, validate existing tests, or improve test coverage. This includes unit tests, integration tests, end-to-end tests, and test architecture decisions. <example>\nContext: The user has just written a new API endpoint and wants to ensure it has proper test coverage.\nuser: "I've created a new endpoint for user authentication. Can you help me test it?"\nassistant: "I'll use the qa-testing-specialist agent to create a comprehensive testing strategy for your authentication endpoint."\n<commentary>\nSince the user needs help with testing their new endpoint, use the qa-testing-specialist agent to analyze the code and create appropriate test cases.\n</commentary>\n</example>\n<example>\nContext: The user wants to improve test coverage for their existing codebase.\nuser: "Our test coverage is at 65%. How can we improve it?"\nassistant: "Let me use the qa-testing-specialist agent to analyze your test coverage and identify areas for improvement."\n<commentary>\nThe user is asking about test coverage improvement, which is a core QA task, so use the qa-testing-specialist agent.\n</commentary>\n</example>
|
||||
color: yellow
|
||||
---
|
||||
|
||||
You are an elite QA Testing Specialist with deep expertise in software quality assurance, test automation, and validation strategies. Your mission is to ensure code quality through comprehensive testing approaches that catch bugs early and maintain high reliability standards.
|
||||
|
||||
**Core Responsibilities:**
|
||||
|
||||
You will analyze code and testing requirements to:
|
||||
- Design comprehensive test strategies covering unit, integration, and end-to-end testing
|
||||
- Write clear, maintainable test cases that validate both happy paths and edge cases
|
||||
- Identify gaps in existing test coverage and propose improvements
|
||||
- Review test code for best practices and maintainability
|
||||
- Suggest appropriate testing frameworks and tools based on the technology stack
|
||||
- Create test data strategies and mock/stub implementations
|
||||
- Validate that tests are actually testing meaningful behavior, not just implementation details
|
||||
|
||||
**Testing Methodology:**
|
||||
|
||||
When analyzing code for testing:
|
||||
1. First understand the business logic and user requirements
|
||||
2. Identify all possible execution paths and edge cases
|
||||
3. Determine the appropriate testing pyramid balance (unit vs integration vs e2e)
|
||||
4. Consider both positive and negative test scenarios
|
||||
5. Ensure tests are isolated, repeatable, and fast
|
||||
6. Validate error handling and boundary conditions
|
||||
|
||||
For test creation:
|
||||
- Write descriptive test names that explain what is being tested and expected behavior
|
||||
- Follow AAA pattern (Arrange, Act, Assert) or Given-When-Then structure
|
||||
- Keep tests focused on single behaviors
|
||||
- Use appropriate assertions that clearly communicate intent
|
||||
- Include setup and teardown when necessary
|
||||
- Consider performance implications of test suites
|
||||
|
||||
**Quality Standards:**
|
||||
|
||||
You will ensure tests:
|
||||
- Are deterministic and don't rely on external state
|
||||
- Run quickly and can be executed in parallel when possible
|
||||
- Provide clear failure messages that help diagnose issues
|
||||
- Cover critical business logic thoroughly
|
||||
- Include regression tests for previously found bugs
|
||||
- Are maintainable and refactorable alongside production code
|
||||
|
||||
**Technology Considerations:**
|
||||
|
||||
Adapt your recommendations based on the project stack. For this codebase using Bun, SQLite, and React:
|
||||
- Leverage Bun's native test runner for JavaScript/TypeScript tests
|
||||
- Consider SQLite in-memory databases for integration tests
|
||||
- Suggest React Testing Library patterns for component testing
|
||||
- Recommend API testing strategies for Astro endpoints
|
||||
- Propose mocking strategies for external services (GitHub/Gitea APIs)
|
||||
|
||||
**Communication Style:**
|
||||
|
||||
You will:
|
||||
- Explain testing decisions with clear rationale
|
||||
- Provide code examples that demonstrate best practices
|
||||
- Prioritize test recommendations based on risk and value
|
||||
- Use precise technical language while remaining accessible
|
||||
- Highlight potential issues proactively
|
||||
- Suggest incremental improvements for existing test suites
|
||||
|
||||
**Edge Case Handling:**
|
||||
|
||||
When encountering:
|
||||
- Legacy code without tests: Propose a pragmatic approach to add tests incrementally
|
||||
- Complex dependencies: Recommend appropriate mocking/stubbing strategies
|
||||
- Performance concerns: Balance thoroughness with execution speed
|
||||
- Flaky tests: Identify root causes and suggest stabilization techniques
|
||||
- Missing requirements: Ask clarifying questions to understand expected behavior
|
||||
|
||||
Your goal is to elevate code quality through strategic testing that builds confidence in the software while maintaining development velocity. Focus on tests that provide maximum value and catch real issues rather than achieving arbitrary coverage metrics.
|
||||
@@ -1,68 +0,0 @@
|
||||
---
|
||||
name: senior-code-architect
|
||||
description: Use this agent when you need to write new code, refactor existing code, implement features, or architect solutions that require deep understanding of software engineering principles and the project's tech stack (Astro, React, Tailwind, Better Auth, Shadcn). This includes creating components, API endpoints, database queries, authentication flows, and ensuring code follows established patterns from CLAUDE.md. Examples:\n\n<example>\nContext: The user needs to implement a new feature or component.\nuser: "Create a new dashboard component that shows repository statistics"\nassistant: "I'll use the senior-code-architect agent to design and implement this dashboard component following the project's patterns."\n<commentary>\nSince this requires creating new code with the project's tech stack, the senior-code-architect agent is appropriate.\n</commentary>\n</example>\n\n<example>\nContext: The user wants to refactor or improve existing code.\nuser: "Refactor the authentication flow to be more maintainable"\nassistant: "Let me use the senior-code-architect agent to analyze and refactor the authentication flow."\n<commentary>\nThis requires deep understanding of Better Auth and clean code principles, making the senior-code-architect agent the right choice.\n</commentary>\n</example>\n\n<example>\nContext: After writing code, the user might want it reviewed.\nuser: "I just implemented the mirror scheduling feature"\nassistant: "Great! Now I'll use the senior-code-architect agent to review the implementation and suggest any improvements."\n<commentary>\nThe senior-code-architect can review recently written code for best practices and design patterns.\n</commentary>\n</example>
|
||||
color: cyan
|
||||
---
|
||||
|
||||
You are a senior software engineer with deep expertise in modern web development, specializing in the Astro + React + Tailwind CSS + Better Auth + Shadcn UI stack. You have extensive experience building scalable, maintainable applications and are known for writing clean, efficient code that follows SOLID principles and established design patterns.
|
||||
|
||||
**Your Core Responsibilities:**
|
||||
|
||||
1. **Write Production-Quality Code**: Create clean, maintainable, and efficient code that follows the project's established patterns from CLAUDE.md. Always use TypeScript for type safety.
|
||||
|
||||
2. **Follow Project Architecture**: Adhere strictly to the project structure:
|
||||
- API endpoints in `/src/pages/api/[resource]/[action].ts` using `createSecureErrorResponse` for error handling
|
||||
- Database queries in `/src/lib/db/queries/` organized by domain
|
||||
- React components in `/src/components/[feature]/` using Shadcn UI components
|
||||
- Custom hooks in `/src/hooks/` for data fetching
|
||||
|
||||
3. **Implement Best Practices**:
|
||||
- Use composition over inheritance
|
||||
- Apply DRY (Don't Repeat Yourself) principles
|
||||
- Write self-documenting code with clear variable and function names
|
||||
- Implement proper error handling and validation
|
||||
- Ensure code is testable and maintainable
|
||||
|
||||
4. **Technology-Specific Guidelines**:
|
||||
- **Astro**: Use SSR capabilities effectively, implement proper API routes
|
||||
- **React**: Use functional components with hooks, implement proper state management
|
||||
- **Tailwind CSS v4**: Use utility classes efficiently, follow the project's styling patterns
|
||||
- **Better Auth**: Implement secure authentication flows, use session validation properly
|
||||
- **Shadcn UI**: Leverage existing components, maintain consistent UI patterns
|
||||
- **Drizzle ORM**: Write efficient database queries, use proper schema definitions
|
||||
|
||||
5. **Code Review Approach**: When reviewing code:
|
||||
- Check for adherence to project patterns and CLAUDE.md guidelines
|
||||
- Identify potential performance issues or bottlenecks
|
||||
- Suggest improvements for readability and maintainability
|
||||
- Ensure proper error handling and edge case coverage
|
||||
- Verify security best practices are followed
|
||||
|
||||
6. **Problem-Solving Methodology**:
|
||||
- Analyze requirements thoroughly before coding
|
||||
- Break down complex problems into smaller, manageable pieces
|
||||
- Consider edge cases and error scenarios
|
||||
- Optimize for both performance and maintainability
|
||||
- Document complex logic with clear comments
|
||||
|
||||
7. **Quality Assurance**:
|
||||
- Write code that is easy to test
|
||||
- Consider adding appropriate test cases using Bun's test runner
|
||||
- Validate inputs and handle errors gracefully
|
||||
- Ensure code works across different scenarios
|
||||
|
||||
**Output Guidelines**:
|
||||
- Provide complete, working code implementations
|
||||
- Include clear explanations of design decisions
|
||||
- Suggest tests when appropriate
|
||||
- Highlight any potential issues or areas for future improvement
|
||||
- Follow the existing code style and conventions
|
||||
|
||||
**Important Reminders**:
|
||||
- Never create files unless absolutely necessary
|
||||
- Always prefer editing existing files
|
||||
- Don't create documentation unless explicitly requested
|
||||
- Focus on the specific task at hand
|
||||
- Reference CLAUDE.md for project-specific patterns and guidelines
|
||||
|
||||
You approach every task with the mindset of a seasoned engineer who values code quality, maintainability, and long-term project health. Your solutions should be elegant, efficient, and aligned with the project's established patterns.
|
||||
@@ -1,61 +0,0 @@
|
||||
---
|
||||
name: strategic-task-planner
|
||||
description: Use this agent when you need to decompose complex projects, features, or problems into structured, actionable plans. This includes breaking down large development tasks, creating implementation roadmaps, organizing multi-step processes, or planning project phases. The agent excels at identifying dependencies, sequencing tasks, and creating clear execution strategies. <example>Context: User needs help planning the implementation of a new feature. user: "I need to add a bulk import feature that can handle CSV files with 100k+ rows" assistant: "I'll use the strategic-task-planner agent to break this down into manageable components and create an implementation plan." <commentary>Since the user is asking about implementing a complex feature, use the Task tool to launch the strategic-task-planner agent to decompose it into actionable steps.</commentary></example> <example>Context: User wants to refactor a large codebase. user: "We need to migrate our entire authentication system from sessions to JWT tokens" assistant: "Let me use the strategic-task-planner agent to create a phased migration plan that minimizes risk." <commentary>Since this is a complex migration requiring careful planning, use the strategic-task-planner agent to create a structured approach.</commentary></example>
|
||||
tools: Glob, Grep, LS, ExitPlanMode, Read, NotebookRead, WebFetch, TodoWrite, WebSearch, Task, mcp__ide__getDiagnostics, mcp__ide__executeCode, mcp__playwright__browser_close, mcp__playwright__browser_resize, mcp__playwright__browser_console_messages, mcp__playwright__browser_handle_dialog, mcp__playwright__browser_evaluate, mcp__playwright__browser_file_upload, mcp__playwright__browser_install, mcp__playwright__browser_press_key, mcp__playwright__browser_type, mcp__playwright__browser_navigate, mcp__playwright__browser_navigate_back, mcp__playwright__browser_navigate_forward, mcp__playwright__browser_network_requests, mcp__playwright__browser_take_screenshot, mcp__playwright__browser_snapshot, mcp__playwright__browser_click, mcp__playwright__browser_drag, mcp__playwright__browser_hover, mcp__playwright__browser_select_option, mcp__playwright__browser_tab_list, mcp__playwright__browser_tab_new, mcp__playwright__browser_tab_select, mcp__playwright__browser_tab_close, mcp__playwright__browser_wait_for
|
||||
color: blue
|
||||
---
|
||||
|
||||
You are a strategic planning specialist with deep expertise in decomposing complex tasks and creating actionable execution plans. Your role is to transform ambiguous or overwhelming projects into clear, structured roadmaps that teams can confidently execute.
|
||||
|
||||
When analyzing a task or project, you will:
|
||||
|
||||
1. **Understand the Core Objective**: Extract the fundamental goal, success criteria, and constraints. Ask clarifying questions if critical details are missing.
|
||||
|
||||
2. **Decompose Systematically**: Break down the task using these principles:
|
||||
- Identify major phases or milestones
|
||||
- Decompose each phase into concrete, actionable tasks
|
||||
- Keep tasks small enough to complete in 1-4 hours when possible
|
||||
- Ensure each task has clear completion criteria
|
||||
|
||||
3. **Map Dependencies**: Identify and document:
|
||||
- Task prerequisites and dependencies
|
||||
- Critical path items that could block progress
|
||||
- Parallel work streams that can proceed independently
|
||||
- Resource or knowledge requirements
|
||||
|
||||
4. **Sequence Strategically**: Order tasks by:
|
||||
- Technical dependencies (what must come first)
|
||||
- Risk mitigation (tackle unknowns early)
|
||||
- Value delivery (enable early feedback when possible)
|
||||
- Resource efficiency (batch similar work)
|
||||
|
||||
5. **Provide Actionable Output**: Structure your plans with:
|
||||
- **Phase Overview**: High-level phases with objectives
|
||||
- **Task Breakdown**: Numbered tasks with clear descriptions
|
||||
- **Dependencies**: Explicitly stated prerequisites
|
||||
- **Effort Estimates**: Rough time estimates when relevant
|
||||
- **Risk Considerations**: Potential blockers or challenges
|
||||
- **Success Metrics**: How to measure completion
|
||||
|
||||
6. **Adapt to Context**: Tailor your planning approach based on:
|
||||
- Technical vs non-technical tasks
|
||||
- Team size and skill level
|
||||
- Time constraints and deadlines
|
||||
- Available resources and tools
|
||||
|
||||
**Output Format Guidelines**:
|
||||
- Use clear hierarchical structure (phases → tasks → subtasks)
|
||||
- Number all tasks for easy reference
|
||||
- Bold key terms and phase names
|
||||
- Include time estimates in brackets [2-4 hours]
|
||||
- Mark critical path items with ⚡
|
||||
- Flag high-risk items with ⚠️
|
||||
|
||||
**Quality Checks**:
|
||||
- Ensure no task is too large or vague
|
||||
- Verify all dependencies are identified
|
||||
- Confirm the plan addresses the original objective
|
||||
- Check that success criteria are measurable
|
||||
- Validate that the sequence makes logical sense
|
||||
|
||||
Remember: A good plan reduces uncertainty and builds confidence. Focus on clarity, completeness, and actionability. When in doubt, err on the side of breaking things down further rather than leaving ambiguity.
|
||||
@@ -1,5 +0,0 @@
|
||||
Evaluate all the updates being made.
|
||||
Update CHANGELOG.md
|
||||
Use the chnages in the git log to determine if its a major, minor or a patch release.
|
||||
Update the package.json first before you push the tag.
|
||||
Never mention Claude Code in the release notes or in commit messages.
|
||||
@@ -1,3 +0,0 @@
|
||||
Generate release notes for the latest release.
|
||||
Use a temp md file to write the release notes.
|
||||
Do not check that file into git.
|
||||
@@ -1,8 +0,0 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(docker build:*)"
|
||||
],
|
||||
"deny": []
|
||||
}
|
||||
}
|
||||
11
.github/workflows/docker-build.yml
vendored
11
.github/workflows/docker-build.yml
vendored
@@ -149,7 +149,11 @@ jobs:
|
||||
### Pull and Test
|
||||
\`\`\`bash
|
||||
docker pull ${imagePath}
|
||||
docker run -d -p 3000:3000 --name gitea-mirror-test ${imagePath}
|
||||
docker run -d \
|
||||
-p 4321:4321 \
|
||||
-e BETTER_AUTH_SECRET=your-secret-here \
|
||||
-e BETTER_AUTH_URL=http://localhost:4321 \
|
||||
--name gitea-mirror-test ${imagePath}
|
||||
\`\`\`
|
||||
|
||||
### Docker Compose Testing
|
||||
@@ -158,9 +162,11 @@ jobs:
|
||||
gitea-mirror:
|
||||
image: ${imagePath}
|
||||
ports:
|
||||
- "3000:3000"
|
||||
- "4321:4321"
|
||||
environment:
|
||||
- BETTER_AUTH_SECRET=your-secret-here
|
||||
- BETTER_AUTH_URL=http://localhost:4321
|
||||
- BETTER_AUTH_TRUSTED_ORIGINS=http://localhost:4321
|
||||
\`\`\`
|
||||
|
||||
> 💡 **Note:** PR images are tagged as \`pr-<number>\` and only built for \`linux/amd64\` to speed up CI.
|
||||
@@ -224,4 +230,3 @@ jobs:
|
||||
continue-on-error: true
|
||||
with:
|
||||
sarif_file: scout-results.sarif
|
||||
|
||||
|
||||
459
CLAUDE.md
459
CLAUDE.md
@@ -2,255 +2,316 @@
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
DONT HALLUCIATE THINGS. IF YOU DONT KNOW LOOK AT THE CODE OR ASK FOR DOCS
|
||||
|
||||
NEVER MENTION CLAUDE CODE ANYWHERE.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Gitea Mirror is a web application that automatically mirrors repositories from GitHub to self-hosted Gitea instances. It uses Astro for SSR, React for UI, SQLite for data storage, and Bun as the JavaScript runtime.
|
||||
Gitea Mirror is a self-hosted web application that automatically mirrors repositories from GitHub to Gitea instances. It's built with Astro (SSR mode), React, and runs on the Bun runtime with SQLite for data persistence.
|
||||
|
||||
## Essential Commands
|
||||
**Key capabilities:**
|
||||
- Mirrors public, private, and starred GitHub repos to Gitea
|
||||
- Supports metadata mirroring (issues, PRs as issues, labels, milestones, releases, wiki)
|
||||
- Git LFS support
|
||||
- Multiple authentication methods (email/password, OIDC/SSO, header auth)
|
||||
- Scheduled automatic syncing with configurable intervals
|
||||
- Auto-discovery of new repos and cleanup of deleted repos
|
||||
- Multi-user support with encrypted token storage (AES-256-GCM)
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Setup and Installation
|
||||
```bash
|
||||
# Install dependencies
|
||||
bun install
|
||||
|
||||
# Initialize database (first time setup)
|
||||
bun run setup
|
||||
|
||||
# Clean start (reset database)
|
||||
bun run dev:clean
|
||||
```
|
||||
|
||||
### Development
|
||||
```bash
|
||||
bun run dev # Start development server (port 3000)
|
||||
bun run build # Build for production
|
||||
bun run preview # Preview production build
|
||||
# Start development server (http://localhost:4321)
|
||||
bun run dev
|
||||
|
||||
# Build for production
|
||||
bun run build
|
||||
|
||||
# Preview production build
|
||||
bun run preview
|
||||
|
||||
# Start production server
|
||||
bun run start
|
||||
```
|
||||
|
||||
### Testing
|
||||
```bash
|
||||
bun test # Run all tests
|
||||
bun test:watch # Run tests in watch mode
|
||||
bun test:coverage # Run tests with coverage
|
||||
# Run all tests
|
||||
bun test
|
||||
|
||||
# Run tests in watch mode
|
||||
bun test:watch
|
||||
|
||||
# Run tests with coverage
|
||||
bun test:coverage
|
||||
```
|
||||
|
||||
**Test configuration:**
|
||||
- Test runner: Bun's built-in test runner (configured in `bunfig.toml`)
|
||||
- Setup file: `src/tests/setup.bun.ts` (auto-loaded via bunfig.toml)
|
||||
- Timeout: 5000ms default
|
||||
- Tests are colocated with source files using `*.test.ts` pattern
|
||||
|
||||
### Database Management
|
||||
```bash
|
||||
bun run init-db # Initialize database
|
||||
bun run reset-users # Reset user accounts (development)
|
||||
bun run cleanup-db # Remove database files
|
||||
# Database operations via Drizzle
|
||||
bun run db:generate # Generate migrations from schema
|
||||
bun run db:migrate # Run migrations
|
||||
bun run db:push # Push schema changes directly
|
||||
bun run db:studio # Open Drizzle Studio (database GUI)
|
||||
bun run db:check # Check schema consistency
|
||||
|
||||
# Database utilities via custom scripts
|
||||
bun run manage-db init # Initialize database
|
||||
bun run manage-db check # Check database health
|
||||
bun run manage-db fix # Fix database issues
|
||||
bun run manage-db reset-users # Reset all users
|
||||
bun run cleanup-db # Delete database file
|
||||
```
|
||||
|
||||
### Production
|
||||
### Utility Scripts
|
||||
```bash
|
||||
bun run start # Start production server
|
||||
# Recovery and diagnostic scripts
|
||||
bun run startup-recovery # Recover from crashes
|
||||
bun run startup-recovery-force # Force recovery
|
||||
bun run test-recovery # Test recovery mechanism
|
||||
bun run test-shutdown # Test graceful shutdown
|
||||
|
||||
# Environment configuration
|
||||
bun run startup-env-config # Load config from env vars
|
||||
```
|
||||
|
||||
## Architecture & Key Concepts
|
||||
## Architecture
|
||||
|
||||
### Technology Stack
|
||||
- **Frontend**: Astro (SSR) + React + Tailwind CSS v4 + Shadcn UI
|
||||
- **Backend**: Bun runtime + SQLite + Drizzle ORM
|
||||
- **APIs**: GitHub (Octokit) and Gitea APIs
|
||||
- **Auth**: Better Auth with email/password, SSO, and OIDC provider support
|
||||
### Tech Stack
|
||||
- **Frontend:** Astro v5 (SSR mode) + React v19 + Shadcn UI + Tailwind CSS v4
|
||||
- **Backend:** Astro API routes (Node adapter, standalone mode)
|
||||
- **Runtime:** Bun (>=1.2.9)
|
||||
- **Database:** SQLite via Drizzle ORM
|
||||
- **Authentication:** Better Auth (session-based)
|
||||
- **APIs:** GitHub (Octokit with throttling plugin), Gitea REST API
|
||||
|
||||
### Project Structure
|
||||
- `/src/pages/api/` - API endpoints (Astro API routes)
|
||||
- `/src/components/` - React components organized by feature
|
||||
- `/src/lib/db/` - Database queries and schema (Drizzle ORM)
|
||||
- `/src/hooks/` - Custom React hooks for data fetching
|
||||
- `/data/` - SQLite database storage location
|
||||
### Directory Structure
|
||||
|
||||
```
|
||||
src/
|
||||
├── components/ # React components (UI, features)
|
||||
│ ├── ui/ # Shadcn UI components
|
||||
│ ├── repositories/ # Repository management components
|
||||
│ ├── organizations/ # Organization management components
|
||||
│ └── ...
|
||||
├── pages/ # Astro pages and API routes
|
||||
│ ├── api/ # API endpoints (Better Auth integration)
|
||||
│ │ ├── auth/ # Authentication endpoints
|
||||
│ │ ├── github/ # GitHub operations
|
||||
│ │ ├── gitea/ # Gitea operations
|
||||
│ │ ├── sync/ # Mirror sync operations
|
||||
│ │ ├── job/ # Job management
|
||||
│ │ └── ...
|
||||
│ └── *.astro # Page components
|
||||
├── lib/ # Core business logic
|
||||
│ ├── db/ # Database (Drizzle ORM)
|
||||
│ │ ├── schema.ts # Database schema with Zod validation
|
||||
│ │ ├── index.ts # Database instance and table exports
|
||||
│ │ └── adapter.ts # Better Auth SQLite adapter
|
||||
│ ├── github.ts # GitHub API client (Octokit)
|
||||
│ ├── gitea.ts # Gitea API client
|
||||
│ ├── gitea-enhanced.ts # Enhanced Gitea operations (metadata)
|
||||
│ ├── scheduler-service.ts # Automatic mirroring scheduler
|
||||
│ ├── cleanup-service.ts # Activity log cleanup
|
||||
│ ├── repository-cleanup-service.ts # Orphaned repo cleanup
|
||||
│ ├── auth.ts # Better Auth configuration
|
||||
│ ├── config.ts # Configuration management
|
||||
│ ├── helpers.ts # Mirror job creation
|
||||
│ ├── utils/ # Utility functions
|
||||
│ │ ├── encryption.ts # AES-256-GCM token encryption
|
||||
│ │ ├── config-encryption.ts # Config token encryption
|
||||
│ │ ├── duration-parser.ts # Parse intervals (e.g., "8h", "30m")
|
||||
│ │ ├── concurrency.ts # Concurrency control utilities
|
||||
│ │ └── mirror-strategies.ts # Mirror strategy logic
|
||||
│ └── ...
|
||||
├── types/ # TypeScript type definitions
|
||||
├── tests/ # Test utilities and setup
|
||||
└── middleware.ts # Astro middleware (auth, session)
|
||||
|
||||
scripts/ # Utility scripts
|
||||
├── manage-db.ts # Database management CLI
|
||||
├── startup-recovery.ts # Crash recovery
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Key Architectural Patterns
|
||||
|
||||
1. **API Routes**: All API endpoints follow the pattern `/api/[resource]/[action]` and use `createSecureErrorResponse` for consistent error handling:
|
||||
```typescript
|
||||
import { createSecureErrorResponse } from '@/lib/utils/error-handler';
|
||||
#### 1. Database Schema and Validation
|
||||
- **Location:** `src/lib/db/schema.ts`
|
||||
- **Pattern:** Drizzle ORM tables + Zod schemas for validation
|
||||
- **Key tables:**
|
||||
- `configs` - User configuration (GitHub/Gitea settings, mirror options)
|
||||
- `repositories` - Tracked repositories with metadata
|
||||
- `organizations` - GitHub organizations with destination overrides
|
||||
- `mirrorJobs` - Mirror job queue and history
|
||||
- `activities` - Activity log for dashboard
|
||||
- `user`, `session`, `account` - Better Auth tables
|
||||
|
||||
export async function POST({ request }: APIContext) {
|
||||
try {
|
||||
// Implementation
|
||||
} catch (error) {
|
||||
return createSecureErrorResponse(error);
|
||||
}
|
||||
}
|
||||
```
|
||||
**Important:** All config tokens (GitHub/Gitea) are encrypted at rest using AES-256-GCM. Use helper functions from `src/lib/utils/config-encryption.ts` to decrypt.
|
||||
|
||||
2. **Database Queries**: Located in `/src/lib/db/queries/` organized by domain (users, repositories, etc.)
|
||||
#### 2. Mirror Job System
|
||||
- **Location:** `src/lib/helpers.ts` (createMirrorJob)
|
||||
- **Flow:**
|
||||
1. User triggers mirror via API endpoint
|
||||
2. `createMirrorJob()` creates job record with status "pending"
|
||||
3. Job processor (in API routes) performs GitHub → Gitea operations
|
||||
4. Job status updated throughout: "mirroring" → "success"/"failed"
|
||||
5. Events published via SSE for real-time UI updates
|
||||
|
||||
3. **Real-time Updates**: Server-Sent Events (SSE) endpoint at `/api/events` for live dashboard updates
|
||||
#### 3. GitHub ↔ Gitea Mirroring
|
||||
- **GitHub Client:** `src/lib/github.ts` - Octokit with rate limit tracking
|
||||
- **Gitea Client:** `src/lib/gitea.ts` - Basic repo operations
|
||||
- **Enhanced Gitea:** `src/lib/gitea-enhanced.ts` - Metadata mirroring (issues, PRs, releases)
|
||||
|
||||
4. **Authentication System**:
|
||||
- Built on Better Auth library
|
||||
- Three authentication methods:
|
||||
- Email & Password (traditional auth)
|
||||
- SSO (authenticate via external OIDC providers)
|
||||
- OIDC Provider (act as OIDC provider for other apps)
|
||||
- Session-based authentication with secure cookies
|
||||
- First user signup creates admin account
|
||||
- Protected routes use Better Auth session validation
|
||||
**Mirror strategies (configured per user):**
|
||||
- `preserve` - Maintain GitHub org structure in Gitea
|
||||
- `single-org` - All repos into one Gitea org
|
||||
- `flat-user` - All repos under user account
|
||||
- `mixed` - Personal repos in one org, org repos preserve structure
|
||||
|
||||
5. **Mirror Process**:
|
||||
- Discovers repos from GitHub (user/org)
|
||||
- Creates/updates mirror in Gitea
|
||||
- Tracks status in database
|
||||
- Supports scheduled automatic mirroring
|
||||
**Metadata mirroring:**
|
||||
- Issues transferred with comments, labels, assignees
|
||||
- PRs converted to issues (Gitea API limitation - cannot create PRs)
|
||||
- Tagged with "pull-request" label
|
||||
- Title prefixed with `[PR #number] [STATUS]`
|
||||
- Body includes commit history, file changes, merge status
|
||||
- Releases mirrored with assets
|
||||
- Labels and milestones preserved
|
||||
- Wiki content cloned if enabled
|
||||
- **Sequential processing:** Issues/PRs mirrored one at a time to prevent out-of-order creation (see `src/lib/gitea-enhanced.ts`)
|
||||
|
||||
6. **Mirror Strategies**: Four ways to organize repositories in Gitea:
|
||||
- **preserve**: Maintains GitHub structure (default)
|
||||
- Organization repos → Same organization name in Gitea
|
||||
- Personal repos → Under your Gitea username
|
||||
- **single-org**: All repos go to one organization
|
||||
- All repos → Single configured organization
|
||||
- **flat-user**: All repos go under user account
|
||||
- All repos → Under your Gitea username
|
||||
- **mixed**: Hybrid approach
|
||||
- Organization repos → Preserve structure
|
||||
- Personal repos → Single configured organization
|
||||
- Starred repos always go to separate organization (starredReposOrg, default: "starred")
|
||||
- Routing logic in `getGiteaRepoOwner()` function
|
||||
#### 4. Scheduler Service
|
||||
- **Location:** `src/lib/scheduler-service.ts`
|
||||
- **Features:**
|
||||
- Cron-based or interval-based scheduling (uses `duration-parser.ts`)
|
||||
- Auto-start on boot when `SCHEDULE_ENABLED=true` or `GITEA_MIRROR_INTERVAL` is set
|
||||
- Auto-import new GitHub repos
|
||||
- Auto-cleanup orphaned repos (archive or delete)
|
||||
- Respects per-repo mirror intervals (not Gitea's default 24h)
|
||||
- **Concurrency control:** Uses `src/lib/utils/concurrency.ts` for batch processing
|
||||
|
||||
### Database Schema (SQLite)
|
||||
- `users` - User accounts and authentication
|
||||
- `configs` - GitHub/Gitea connection settings
|
||||
- `repositories` - Repository mirror status and metadata
|
||||
- `organizations` - Organization structure preservation
|
||||
- `mirror_jobs` - Scheduled mirror operations
|
||||
- `events` - Activity log and notifications
|
||||
#### 5. Authentication System
|
||||
- **Location:** `src/lib/auth.ts`, `src/lib/auth-client.ts`
|
||||
- **Better Auth integration:**
|
||||
- Email/password (always enabled)
|
||||
- OIDC/SSO providers (configurable via UI)
|
||||
- Header authentication for reverse proxies (Authentik, Authelia)
|
||||
- **Session management:** Cookie-based, validated in Astro middleware
|
||||
- **User helpers:** `src/lib/utils/auth-helpers.ts`
|
||||
|
||||
### Testing Approach
|
||||
- Uses Bun's native test runner (`bun:test`)
|
||||
- Test files use `.test.ts` or `.test.tsx` extension
|
||||
- Setup file at `/src/tests/setup.bun.ts`
|
||||
- Mock utilities available for API testing.
|
||||
#### 6. Environment Configuration
|
||||
- **Startup:** `src/lib/env-config-loader.ts` + `scripts/startup-env-config.ts`
|
||||
- **Pattern:** Environment variables can pre-configure settings, but users can override via web UI
|
||||
- **Encryption:** `ENCRYPTION_SECRET` for tokens, `BETTER_AUTH_SECRET` for sessions
|
||||
|
||||
### Development Tips
|
||||
- Environment variables in `.env` (copy from `.env.example`)
|
||||
- BETTER_AUTH_SECRET required for session signing
|
||||
- Database auto-initializes on first run
|
||||
- Use `bun run dev:clean` for fresh database start
|
||||
- Tailwind CSS v4 configured with Vite plugin
|
||||
#### 7. Real-time Updates
|
||||
- **Events:** `src/lib/events.ts` + `src/lib/events/realtime.ts`
|
||||
- **Pattern:** Server-Sent Events (SSE) for live dashboard updates
|
||||
- **Endpoints:** `/api/sse` - client subscribes to job/repo events
|
||||
|
||||
### Authentication Setup
|
||||
- **Better Auth** handles all authentication
|
||||
- Configuration in `/src/lib/auth.ts` (server) and `/src/lib/auth-client.ts` (client)
|
||||
- Auth endpoints available at `/api/auth/*`
|
||||
- SSO providers configured through the web UI
|
||||
- OIDC provider functionality for external applications
|
||||
### Testing Patterns
|
||||
|
||||
### Common Tasks
|
||||
**Unit tests:**
|
||||
- Colocated with source: `filename.test.ts` alongside `filename.ts`
|
||||
- Use Bun's built-in assertions and mocking
|
||||
- Mock external APIs (GitHub, Gitea) using `src/tests/mock-fetch.ts`
|
||||
|
||||
**Adding a new API endpoint:**
|
||||
1. Create file in `/src/pages/api/[resource]/[action].ts`
|
||||
2. Use `createSecureErrorResponse` for error handling
|
||||
3. Add corresponding database query in `/src/lib/db/queries/`
|
||||
4. Update types in `/src/types/` if needed
|
||||
**Integration tests:**
|
||||
- Located in `src/tests/`
|
||||
- Test database operations with in-memory SQLite
|
||||
- Example: `src/lib/db/index.test.ts`
|
||||
|
||||
**Adding a new component:**
|
||||
1. Create in appropriate `/src/components/[feature]/` directory
|
||||
2. Use Shadcn UI components from `/src/components/ui/`
|
||||
3. Follow existing naming patterns (e.g., `RepositoryCard`, `ConfigTabs`)
|
||||
**Test utilities:**
|
||||
- `src/tests/setup.bun.ts` - Global test setup (loaded via bunfig.toml)
|
||||
- `src/tests/mock-fetch.ts` - Fetch mocking utilities
|
||||
|
||||
**Modifying database schema:**
|
||||
1. Update schema in `/src/lib/db/schema.ts`
|
||||
2. Run `bun run init-db` to recreate database
|
||||
3. Update related queries in `/src/lib/db/queries/`
|
||||
### Important Development Notes
|
||||
|
||||
## Configuration Options
|
||||
1. **Path Aliases:** Use `@/` for imports (configured in `tsconfig.json`)
|
||||
```typescript
|
||||
import { db } from '@/lib/db';
|
||||
```
|
||||
|
||||
### GitHub Configuration (UI Fields)
|
||||
2. **Token Encryption:** Always use encryption helpers when dealing with tokens:
|
||||
```typescript
|
||||
import { getDecryptedGitHubToken, getDecryptedGiteaToken } from '@/lib/utils/config-encryption';
|
||||
```
|
||||
|
||||
#### Basic Settings (`githubConfig`)
|
||||
- **username**: GitHub username
|
||||
- **token**: GitHub personal access token (requires repo and admin:org scopes)
|
||||
- **privateRepositories**: Include private repositories
|
||||
- **mirrorStarred**: Mirror starred repositories
|
||||
3. **API Route Pattern:** Astro API routes in `src/pages/api/` should:
|
||||
- Check authentication via Better Auth
|
||||
- Validate input with Zod schemas
|
||||
- Handle errors gracefully
|
||||
- Return JSON responses
|
||||
|
||||
### Gitea Configuration (UI Fields)
|
||||
- **url**: Gitea instance URL
|
||||
- **username**: Gitea username
|
||||
- **token**: Gitea access token
|
||||
- **organization**: Destination organization (for single-org/mixed strategies)
|
||||
- **starredReposOrg**: Organization for starred repositories (default: "starred")
|
||||
- **visibility**: Organization visibility - "public", "private", "limited"
|
||||
- **mirrorStrategy**: Repository organization strategy (set via UI)
|
||||
- **preserveOrgStructure**: Automatically set based on mirrorStrategy
|
||||
4. **Database Migrations:**
|
||||
- Schema changes: Update `src/lib/db/schema.ts`
|
||||
- Generate migration: `bun run db:generate`
|
||||
- Review generated SQL in `drizzle/` directory
|
||||
- Apply: `bun run db:migrate` (or `db:push` for dev)
|
||||
|
||||
### Schedule Configuration (`scheduleConfig`)
|
||||
- **enabled**: Enable automatic mirroring (default: false)
|
||||
- **interval**: Cron expression or seconds (default: "0 2 * * *" - 2 AM daily)
|
||||
- **concurrent**: Allow concurrent mirror operations (default: false)
|
||||
- **batchSize**: Number of repos to process in parallel (default: 10)
|
||||
5. **Concurrency Control:**
|
||||
- Use utilities from `src/lib/utils/concurrency.ts` for batch operations
|
||||
- Respect rate limits (GitHub: 5000 req/hr authenticated, Gitea: varies)
|
||||
- Issue/PR mirroring is sequential to maintain chronological order
|
||||
|
||||
### Database Cleanup Configuration (`cleanupConfig`)
|
||||
- **enabled**: Enable automatic cleanup (default: false)
|
||||
- **retentionDays**: Days to keep events (stored as seconds internally)
|
||||
6. **Duration Parsing:**
|
||||
- Use `parseInterval()` from `src/lib/utils/duration-parser.ts`
|
||||
- Supports: "30m", "8h", "24h", "7d", cron expressions, or milliseconds
|
||||
|
||||
### Mirror Options (UI Fields)
|
||||
- **mirrorReleases**: Mirror GitHub releases to Gitea
|
||||
- **mirrorLFS**: Mirror Git LFS (Large File Storage) objects
|
||||
- Requires LFS enabled on Gitea server (LFS_START_SERVER = true)
|
||||
- Requires Git v2.1.2+ on server
|
||||
- **mirrorMetadata**: Enable metadata mirroring (master toggle)
|
||||
- **metadataComponents** (only available when mirrorMetadata is enabled):
|
||||
- **issues**: Mirror issues
|
||||
- **pullRequests**: Mirror pull requests
|
||||
- **labels**: Mirror labels
|
||||
- **milestones**: Mirror milestones
|
||||
- **wiki**: Mirror wiki content
|
||||
7. **Graceful Shutdown:**
|
||||
- Services implement cleanup handlers (see `src/lib/shutdown-manager.ts`)
|
||||
- Recovery system in `src/lib/recovery.ts` handles interrupted jobs
|
||||
|
||||
### Advanced Options (UI Fields)
|
||||
- **skipForks**: Skip forked repositories (default: false)
|
||||
- **starredCodeOnly**: Skip issues for starred repositories (default: false) - enables "Lightweight mode" for starred repos
|
||||
## Common Development Workflows
|
||||
|
||||
### Repository Statuses
|
||||
Repositories can have the following statuses:
|
||||
- **imported**: Repository discovered from GitHub
|
||||
- **mirroring**: Currently being mirrored to Gitea
|
||||
- **mirrored**: Successfully mirrored
|
||||
- **syncing**: Repository being synchronized
|
||||
- **synced**: Successfully synchronized
|
||||
- **failed**: Mirror/sync operation failed
|
||||
- **skipped**: Skipped due to filters or conditions
|
||||
- **ignored**: User explicitly marked to ignore (won't be mirrored/synced)
|
||||
- **deleting**: Repository being deleted
|
||||
- **deleted**: Repository deleted
|
||||
### Adding a new mirror option
|
||||
1. Update Zod schema in `src/lib/db/schema.ts` (e.g., `giteaConfigSchema`)
|
||||
2. Update TypeScript types in `src/types/config.ts`
|
||||
3. Add UI control in settings page component
|
||||
4. Update API handler in `src/pages/api/config/`
|
||||
5. Implement logic in `src/lib/gitea.ts` or `src/lib/gitea-enhanced.ts`
|
||||
|
||||
### Scheduling and Synchronization (Issue #72 Fixes)
|
||||
### Debugging mirror failures
|
||||
1. Check mirror jobs: `bun run db:studio` → `mirrorJobs` table
|
||||
2. Review activity logs: Dashboard → Activity tab
|
||||
3. Check console logs for API errors (GitHub/Gitea rate limits, auth issues)
|
||||
4. Use diagnostic scripts: `bun run test-recovery`
|
||||
|
||||
#### Fixed Issues
|
||||
1. **Mirror Interval Bug**: Added `mirror_interval` parameter to Gitea API calls when creating mirrors (previously defaulted to 24h)
|
||||
2. **Auto-Discovery**: Scheduler now automatically discovers and imports new GitHub repositories
|
||||
3. **Interval Updates**: Sync operations now update existing mirrors' intervals to match configuration
|
||||
4. **Repository Cleanup**: Integrated automatic cleanup of orphaned repositories (repos removed from GitHub)
|
||||
### Adding authentication provider
|
||||
1. Update Better Auth config in `src/lib/auth.ts`
|
||||
2. Add provider configuration UI in settings
|
||||
3. Test with `src/tests/test-gitea-auth.ts` patterns
|
||||
4. Update documentation in `docs/SSO-OIDC-SETUP.md`
|
||||
|
||||
#### Environment Variables for Auto-Import
|
||||
- **AUTO_IMPORT_REPOS**: Set to `false` to disable automatic repository discovery (default: enabled)
|
||||
## Docker Deployment
|
||||
|
||||
#### How Scheduling Works
|
||||
- **Scheduler Service**: Runs every minute to check for scheduled tasks
|
||||
- **Sync Interval**: Configured via `GITEA_MIRROR_INTERVAL` or UI (e.g., "8h", "30m", "1d")
|
||||
- **Auto-Import**: Checks GitHub for new repositories during each scheduled sync
|
||||
- **Auto-Cleanup**: Removes repositories that no longer exist in GitHub (if enabled)
|
||||
- **Mirror Interval Update**: Updates Gitea's internal mirror interval during sync operations
|
||||
- **Dockerfile:** Multi-stage build (bun base → build → production)
|
||||
- **Entrypoint:** `docker-entrypoint.sh` - handles CA certs, user permissions, database init
|
||||
- **Compose files:**
|
||||
- `docker-compose.alt.yml` - Quick start (pre-built image, minimal config)
|
||||
- `docker-compose.yml` - Full setup (build from source, all env vars)
|
||||
- `docker-compose.dev.yml` - Development with hot reload
|
||||
|
||||
### Authentication Configuration
|
||||
## Additional Resources
|
||||
|
||||
#### SSO Provider Configuration
|
||||
- **issuerUrl**: OIDC issuer URL (e.g., https://accounts.google.com)
|
||||
- **domain**: Email domain for this provider
|
||||
- **providerId**: Unique identifier for the provider
|
||||
- **clientId**: OAuth client ID from provider
|
||||
- **clientSecret**: OAuth client secret from provider
|
||||
- **authorizationEndpoint**: OAuth authorization URL (auto-discovered if supported)
|
||||
- **tokenEndpoint**: OAuth token exchange URL (auto-discovered if supported)
|
||||
- **jwksEndpoint**: JSON Web Key Set URL (optional, auto-discovered)
|
||||
- **userInfoEndpoint**: User information endpoint (optional, auto-discovered)
|
||||
|
||||
#### OIDC Provider Settings (for external apps)
|
||||
- **allowedRedirectUris**: Comma-separated list of allowed redirect URIs
|
||||
- **clientId**: Generated client ID for the application
|
||||
- **clientSecret**: Generated client secret for the application
|
||||
- **scopes**: Available scopes (openid, profile, email)
|
||||
|
||||
#### Environment Variables
|
||||
- **BETTER_AUTH_SECRET**: Secret key for signing sessions (required)
|
||||
- **BETTER_AUTH_URL**: Base URL for authentication (default: http://localhost:4321)
|
||||
|
||||
## Security Guidelines
|
||||
|
||||
- **Confidentiality Guidelines**:
|
||||
- Dont ever say Claude Code or generated with AI anyhwere.
|
||||
- Never commit without the explicict ask
|
||||
- **Environment Variables:** See `docs/ENVIRONMENT_VARIABLES.md` for complete list
|
||||
- **Development Workflow:** See `docs/DEVELOPMENT_WORKFLOW.md`
|
||||
- **SSO Setup:** See `docs/SSO-OIDC-SETUP.md`
|
||||
- **Contributing:** See `CONTRIBUTING.md` for code guidelines and scope
|
||||
- **Graceful Shutdown:** See `docs/GRACEFUL_SHUTDOWN.md` for crash recovery details
|
||||
|
||||
@@ -10,10 +10,6 @@
|
||||
</p>
|
||||
</p>
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Upgrading to v3?** v3 requires a fresh start with a new data volume. Please read the [Upgrade Guide](UPGRADE.md) for instructions.
|
||||
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
```bash
|
||||
|
||||
74
UPGRADE.md
74
UPGRADE.md
@@ -1,74 +0,0 @@
|
||||
# Upgrade Guide
|
||||
|
||||
## Upgrading to v3.0
|
||||
|
||||
> **⚠️ IMPORTANT**: v3.0 requires a fresh start. There is no automated migration from v2.x to v3.0.
|
||||
|
||||
### Why No Migration?
|
||||
|
||||
v3.0 introduces fundamental changes to the application architecture:
|
||||
- **Authentication**: Switched from JWT to Better Auth
|
||||
- **Database**: Now uses Drizzle ORM with proper migrations
|
||||
- **Security**: All tokens are now encrypted
|
||||
- **Features**: Added SSO support and OIDC provider functionality
|
||||
|
||||
Due to these extensive changes, we recommend starting fresh with v3.0 for the best experience.
|
||||
|
||||
### Upgrade Steps
|
||||
|
||||
1. **Stop your v2.x container**
|
||||
```bash
|
||||
docker stop gitea-mirror
|
||||
docker rm gitea-mirror
|
||||
```
|
||||
|
||||
2. **Backup your v2.x data (optional)**
|
||||
```bash
|
||||
# If you want to keep your v2 data for reference
|
||||
docker run --rm -v gitea-mirror-data:/data -v $(pwd):/backup alpine tar czf /backup/gitea-mirror-v2-backup.tar.gz -C /data .
|
||||
```
|
||||
|
||||
3. **Create a new volume for v3**
|
||||
```bash
|
||||
docker volume create gitea-mirror-v3-data
|
||||
```
|
||||
|
||||
4. **Run v3 with the new volume**
|
||||
```bash
|
||||
docker run -d \
|
||||
--name gitea-mirror \
|
||||
-p 4321:4321 \
|
||||
-v gitea-mirror-v3-data:/app/data \
|
||||
-e BETTER_AUTH_SECRET=your-secret-key \
|
||||
-e ENCRYPTION_SECRET=your-encryption-key \
|
||||
arunavo4/gitea-mirror:latest
|
||||
```
|
||||
|
||||
5. **Set up your configuration again**
|
||||
- Navigate to http://localhost:4321
|
||||
- Create a new admin account
|
||||
- Re-enter your GitHub and Gitea credentials
|
||||
- Configure your mirror settings
|
||||
|
||||
### What Happens to My Existing Mirrors?
|
||||
|
||||
Your existing mirrors in Gitea are **not affected**. The application will:
|
||||
- Recognize existing repositories when you re-import
|
||||
- Skip creating duplicates
|
||||
- Resume normal mirror operations
|
||||
|
||||
### Environment Variable Changes
|
||||
|
||||
v3.0 uses different environment variables:
|
||||
|
||||
| v2.x | v3.0 | Notes |
|
||||
|------|------|-------|
|
||||
| `JWT_SECRET` | `BETTER_AUTH_SECRET` | Required for session management |
|
||||
| - | `ENCRYPTION_SECRET` | New - required for token encryption |
|
||||
|
||||
### Need Help?
|
||||
|
||||
If you have questions about upgrading:
|
||||
1. Check the [README](README.md) for v3 setup instructions
|
||||
2. Review your v2 configuration before upgrading
|
||||
3. Open an issue if you encounter problems
|
||||
@@ -26,6 +26,10 @@ services:
|
||||
- HOST=0.0.0.0
|
||||
- PORT=4321
|
||||
- PUBLIC_BETTER_AUTH_URL=${PUBLIC_BETTER_AUTH_URL:-http://localhost:4321}
|
||||
# Optional concurrency controls (defaults match in-app defaults)
|
||||
# If you want perfect ordering of issues and PRs, set these at 1
|
||||
- MIRROR_ISSUE_CONCURRENCY=${MIRROR_ISSUE_CONCURRENCY:-3}
|
||||
- MIRROR_PULL_REQUEST_CONCURRENCY=${MIRROR_PULL_REQUEST_CONCURRENCY:-5}
|
||||
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=3", "--spider", "http://localhost:4321/api/health"]
|
||||
@@ -54,4 +58,4 @@ services:
|
||||
# - Auto-import settings
|
||||
# - Cleanup preferences
|
||||
#
|
||||
# That's it! Everything else can be configured via the web interface.
|
||||
# That's it! Everything else can be configured via the web interface.
|
||||
|
||||
@@ -47,6 +47,8 @@ services:
|
||||
- PRESERVE_ORG_STRUCTURE=${PRESERVE_ORG_STRUCTURE:-false}
|
||||
- ONLY_MIRROR_ORGS=${ONLY_MIRROR_ORGS:-false}
|
||||
- SKIP_STARRED_ISSUES=${SKIP_STARRED_ISSUES:-false}
|
||||
- MIRROR_ISSUE_CONCURRENCY=${MIRROR_ISSUE_CONCURRENCY:-3}
|
||||
- MIRROR_PULL_REQUEST_CONCURRENCY=${MIRROR_PULL_REQUEST_CONCURRENCY:-5}
|
||||
- GITEA_URL=${GITEA_URL:-}
|
||||
- GITEA_TOKEN=${GITEA_TOKEN:-}
|
||||
- GITEA_USERNAME=${GITEA_USERNAME:-}
|
||||
|
||||
@@ -141,6 +141,10 @@ Control what content gets mirrored from GitHub to Gitea.
|
||||
| `MIRROR_PULL_REQUESTS` | Mirror pull requests (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
|
||||
| `MIRROR_LABELS` | Mirror labels (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
|
||||
| `MIRROR_MILESTONES` | Mirror milestones (requires MIRROR_METADATA=true) | `false` | `true`, `false` |
|
||||
| `MIRROR_ISSUE_CONCURRENCY` | Number of issues processed in parallel. Set above `1` to speed up mirroring at the risk of out-of-order creation. | `3` | Integer ≥ 1 |
|
||||
| `MIRROR_PULL_REQUEST_CONCURRENCY` | Number of pull requests processed in parallel. Values above `1` may cause ordering differences. | `5` | Integer ≥ 1 |
|
||||
|
||||
> **Ordering vs Throughput:** Metadata now mirrors sequentially by default to preserve chronology. Increase the concurrency variables only if you can tolerate minor out-of-order entries.
|
||||
|
||||
## Automation Configuration
|
||||
|
||||
|
||||
@@ -54,6 +54,8 @@ export const giteaConfigSchema = z.object({
|
||||
.enum(["skip", "reference", "full-copy"])
|
||||
.default("reference"),
|
||||
// Mirror options
|
||||
issueConcurrency: z.number().int().min(1).default(3),
|
||||
pullRequestConcurrency: z.number().int().min(1).default(5),
|
||||
mirrorReleases: z.boolean().default(false),
|
||||
releaseLimit: z.number().default(10),
|
||||
mirrorMetadata: z.boolean().default(false),
|
||||
|
||||
@@ -49,6 +49,9 @@ interface EnvConfig {
|
||||
mirrorLabels?: boolean;
|
||||
mirrorMilestones?: boolean;
|
||||
mirrorMetadata?: boolean;
|
||||
releaseLimit?: number;
|
||||
issueConcurrency?: number;
|
||||
pullRequestConcurrency?: number;
|
||||
};
|
||||
schedule: {
|
||||
enabled?: boolean;
|
||||
@@ -136,6 +139,8 @@ function parseEnvConfig(): EnvConfig {
|
||||
mirrorMilestones: process.env.MIRROR_MILESTONES === 'true',
|
||||
mirrorMetadata: process.env.MIRROR_METADATA === 'true',
|
||||
releaseLimit: process.env.RELEASE_LIMIT ? parseInt(process.env.RELEASE_LIMIT, 10) : undefined,
|
||||
issueConcurrency: process.env.MIRROR_ISSUE_CONCURRENCY ? parseInt(process.env.MIRROR_ISSUE_CONCURRENCY, 10) : undefined,
|
||||
pullRequestConcurrency: process.env.MIRROR_PULL_REQUEST_CONCURRENCY ? parseInt(process.env.MIRROR_PULL_REQUEST_CONCURRENCY, 10) : undefined,
|
||||
},
|
||||
schedule: {
|
||||
enabled: process.env.SCHEDULE_ENABLED === 'true' ||
|
||||
@@ -277,6 +282,12 @@ export async function initializeConfigFromEnv(): Promise<void> {
|
||||
// Mirror metadata options
|
||||
mirrorReleases: envConfig.mirror.mirrorReleases ?? existingConfig?.[0]?.giteaConfig?.mirrorReleases ?? false,
|
||||
releaseLimit: envConfig.mirror.releaseLimit ?? existingConfig?.[0]?.giteaConfig?.releaseLimit ?? 10,
|
||||
issueConcurrency: envConfig.mirror.issueConcurrency && envConfig.mirror.issueConcurrency > 0
|
||||
? envConfig.mirror.issueConcurrency
|
||||
: existingConfig?.[0]?.giteaConfig?.issueConcurrency ?? 3,
|
||||
pullRequestConcurrency: envConfig.mirror.pullRequestConcurrency && envConfig.mirror.pullRequestConcurrency > 0
|
||||
? envConfig.mirror.pullRequestConcurrency
|
||||
: existingConfig?.[0]?.giteaConfig?.pullRequestConcurrency ?? 5,
|
||||
mirrorMetadata: envConfig.mirror.mirrorMetadata ?? (envConfig.mirror.mirrorIssues || envConfig.mirror.mirrorPullRequests || envConfig.mirror.mirrorLabels || envConfig.mirror.mirrorMilestones) ?? existingConfig?.[0]?.giteaConfig?.mirrorMetadata ?? false,
|
||||
mirrorIssues: envConfig.mirror.mirrorIssues ?? existingConfig?.[0]?.giteaConfig?.mirrorIssues ?? false,
|
||||
mirrorPullRequests: envConfig.mirror.mirrorPullRequests ?? existingConfig?.[0]?.giteaConfig?.mirrorPullRequests ?? false,
|
||||
|
||||
@@ -12,6 +12,7 @@ import { createMirrorJob } from "./helpers";
|
||||
import { db, organizations, repositories } from "./db";
|
||||
import { eq, and } from "drizzle-orm";
|
||||
import { decryptConfigTokens } from "./utils/config-encryption";
|
||||
import { formatDateShort } from "./utils";
|
||||
|
||||
/**
|
||||
* Helper function to get organization configuration including destination override
|
||||
@@ -1558,6 +1559,8 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
@@ -1590,6 +1593,18 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
// Import the processWithRetry function
|
||||
const { processWithRetry } = await import("@/lib/utils/concurrency");
|
||||
|
||||
const rawIssueConcurrency = config.giteaConfig?.issueConcurrency ?? 3;
|
||||
const issueConcurrencyLimit =
|
||||
Number.isFinite(rawIssueConcurrency)
|
||||
? Math.max(1, Math.floor(rawIssueConcurrency))
|
||||
: 1;
|
||||
|
||||
if (issueConcurrencyLimit > 1) {
|
||||
console.warn(
|
||||
`[Issues] Concurrency is set to ${issueConcurrencyLimit}. This may lead to out-of-order issue creation in Gitea but is faster.`
|
||||
);
|
||||
}
|
||||
|
||||
// Process issues in parallel with concurrency control
|
||||
await processWithRetry(
|
||||
filteredIssues,
|
||||
@@ -1632,11 +1647,15 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
.join(", ")} on GitHub.`
|
||||
: "";
|
||||
|
||||
const issueAuthor = issue.user?.login ?? "unknown";
|
||||
const issueCreatedOn = formatDateShort(issue.created_at);
|
||||
const issueOriginHeader = `Originally created by @${issueAuthor} on GitHub${
|
||||
issueCreatedOn ? ` (${issueCreatedOn})` : ""
|
||||
}.`;
|
||||
|
||||
const issuePayload: any = {
|
||||
title: issue.title,
|
||||
body: `Originally created by @${
|
||||
issue.user?.login
|
||||
} on GitHub.${originalAssignees}\n\n${issue.body || ""}`,
|
||||
body: `${issueOriginHeader}${originalAssignees}\n\n${issue.body ?? ""}`,
|
||||
closed: issue.state === "closed",
|
||||
labels: giteaLabelIds,
|
||||
};
|
||||
@@ -1662,15 +1681,30 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
(res) => res.data
|
||||
);
|
||||
|
||||
// Process comments in parallel with concurrency control
|
||||
if (comments.length > 0) {
|
||||
// Ensure comments are applied in chronological order to preserve discussion flow
|
||||
const sortedComments = comments
|
||||
.slice()
|
||||
.sort(
|
||||
(a, b) =>
|
||||
new Date(a.created_at || 0).getTime() -
|
||||
new Date(b.created_at || 0).getTime()
|
||||
);
|
||||
|
||||
// Process comments sequentially to preserve historical ordering
|
||||
if (sortedComments.length > 0) {
|
||||
await processWithRetry(
|
||||
comments,
|
||||
sortedComments,
|
||||
async (comment) => {
|
||||
const commenter = comment.user?.login ?? "unknown";
|
||||
const commentDate = formatDateShort(comment.created_at);
|
||||
const commentHeader = `@${commenter} commented on GitHub${
|
||||
commentDate ? ` (${commentDate})` : ""
|
||||
}:`;
|
||||
|
||||
await httpPost(
|
||||
`${config.giteaConfig!.url}/api/v1/repos/${giteaOwner}/${repoName}/issues/${createdIssue.data.number}/comments`,
|
||||
{
|
||||
body: `@${comment.user?.login} commented on GitHub:\n\n${comment.body}`,
|
||||
body: `${commentHeader}\n\n${comment.body ?? ""}`,
|
||||
},
|
||||
{
|
||||
Authorization: `token ${decryptedConfig.giteaConfig!.token}`,
|
||||
@@ -1679,7 +1713,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
return comment;
|
||||
},
|
||||
{
|
||||
concurrencyLimit: 5,
|
||||
concurrencyLimit: 1,
|
||||
maxRetries: 2,
|
||||
retryDelay: 1000,
|
||||
onRetry: (_comment, error, attempt) => {
|
||||
@@ -1694,7 +1728,7 @@ export const mirrorGitRepoIssuesToGitea = async ({
|
||||
return issue;
|
||||
},
|
||||
{
|
||||
concurrencyLimit: 3, // Process 3 issues at a time
|
||||
concurrencyLimit: issueConcurrencyLimit,
|
||||
maxRetries: 2,
|
||||
retryDelay: 2000,
|
||||
onProgress: (completed, total, result) => {
|
||||
@@ -1966,6 +2000,8 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
repo,
|
||||
state: "all",
|
||||
per_page: 100,
|
||||
sort: "created",
|
||||
direction: "asc",
|
||||
},
|
||||
(res) => res.data
|
||||
);
|
||||
@@ -2022,6 +2058,18 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
|
||||
const { processWithRetry } = await import("@/lib/utils/concurrency");
|
||||
|
||||
const rawPullConcurrency = config.giteaConfig?.pullRequestConcurrency ?? 5;
|
||||
const pullRequestConcurrencyLimit =
|
||||
Number.isFinite(rawPullConcurrency)
|
||||
? Math.max(1, Math.floor(rawPullConcurrency))
|
||||
: 1;
|
||||
|
||||
if (pullRequestConcurrencyLimit > 1) {
|
||||
console.warn(
|
||||
`[Pull Requests] Concurrency is set to ${pullRequestConcurrencyLimit}. This may lead to out-of-order pull request mirroring in Gitea.`
|
||||
);
|
||||
}
|
||||
|
||||
let successCount = 0;
|
||||
let failedCount = 0;
|
||||
|
||||
@@ -2144,7 +2192,7 @@ export async function mirrorGitRepoPullRequestsToGitea({
|
||||
}
|
||||
},
|
||||
{
|
||||
concurrencyLimit: 5,
|
||||
concurrencyLimit: pullRequestConcurrencyLimit,
|
||||
maxRetries: 3,
|
||||
retryDelay: 1000,
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, test, expect } from "bun:test";
|
||||
import { jsonResponse, formatDate, truncate, safeParse, parseErrorMessage, showErrorToast } from "./utils";
|
||||
import { jsonResponse, formatDate, formatDateShort, truncate, safeParse, parseErrorMessage, showErrorToast } from "./utils";
|
||||
|
||||
describe("jsonResponse", () => {
|
||||
test("creates a Response with JSON content", () => {
|
||||
@@ -65,6 +65,18 @@ describe("formatDate", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("formatDateShort", () => {
|
||||
test("returns formatted date when input is provided", () => {
|
||||
const formatted = formatDateShort("2014-10-20T15:32:10Z");
|
||||
expect(formatted).toBe("Oct 20, 2014");
|
||||
});
|
||||
|
||||
test("returns undefined when date is missing", () => {
|
||||
expect(formatDateShort(null)).toBeUndefined();
|
||||
expect(formatDateShort(undefined)).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("truncate", () => {
|
||||
test("truncates a string that exceeds the length", () => {
|
||||
const str = "This is a long string that needs truncation";
|
||||
|
||||
@@ -29,6 +29,15 @@ export function formatDate(date?: Date | string | null): string {
|
||||
}).format(new Date(date));
|
||||
}
|
||||
|
||||
export function formatDateShort(date?: Date | string | null): string | undefined {
|
||||
if (!date) return undefined;
|
||||
return new Intl.DateTimeFormat("en-US", {
|
||||
year: "numeric",
|
||||
month: "short",
|
||||
day: "numeric",
|
||||
}).format(new Date(date));
|
||||
}
|
||||
|
||||
export function formatLastSyncTime(date: Date | string | null): string {
|
||||
if (!date) return "Never";
|
||||
|
||||
|
||||
@@ -86,6 +86,8 @@ export async function createDefaultConfig({ userId, envOverrides = {} }: Default
|
||||
addTopics: true,
|
||||
preserveVisibility: false,
|
||||
forkStrategy: "reference",
|
||||
issueConcurrency: 3,
|
||||
pullRequestConcurrency: 5,
|
||||
},
|
||||
include: [],
|
||||
exclude: [],
|
||||
|
||||
@@ -89,6 +89,8 @@ export function mapUiToDbConfig(
|
||||
forkStrategy: advancedOptions.skipForks ? "skip" : "reference",
|
||||
|
||||
// Mirror options from UI
|
||||
issueConcurrency: giteaConfig.issueConcurrency ?? 3,
|
||||
pullRequestConcurrency: giteaConfig.pullRequestConcurrency ?? 5,
|
||||
mirrorReleases: mirrorOptions.mirrorReleases,
|
||||
releaseLimit: mirrorOptions.releaseLimit || 10,
|
||||
mirrorMetadata: mirrorOptions.mirrorMetadata,
|
||||
@@ -132,6 +134,8 @@ export function mapDbToUiConfig(dbConfig: any): {
|
||||
preserveOrgStructure: dbConfig.giteaConfig?.preserveVisibility || false, // Map preserveVisibility
|
||||
mirrorStrategy: dbConfig.githubConfig?.mirrorStrategy || "preserve", // Get from GitHub config
|
||||
personalReposOrg: undefined, // Not stored in current schema
|
||||
issueConcurrency: dbConfig.giteaConfig?.issueConcurrency ?? 3,
|
||||
pullRequestConcurrency: dbConfig.giteaConfig?.pullRequestConcurrency ?? 5,
|
||||
};
|
||||
|
||||
// Map mirror options from various database fields
|
||||
|
||||
@@ -13,6 +13,8 @@ export interface GiteaConfig {
|
||||
preserveOrgStructure: boolean;
|
||||
mirrorStrategy?: MirrorStrategy; // New field for the strategy
|
||||
personalReposOrg?: string; // Override destination for personal repos
|
||||
issueConcurrency?: number;
|
||||
pullRequestConcurrency?: number;
|
||||
}
|
||||
|
||||
export interface ScheduleConfig {
|
||||
|
||||
Reference in New Issue
Block a user