From fd5e68c1d4a84dcc118913d363b4ff3722203c90 Mon Sep 17 00:00:00 2001 From: Arunavo Ray Date: Thu, 23 Oct 2025 05:10:42 +0530 Subject: [PATCH] docs: update development workflow and documentation index Updated development documentation to reflect current project structure and simplified setup process. Changes: - DEVELOPMENT_WORKFLOW.md: Updated repository URL, simplified setup steps, improved project structure documentation, and clarified command descriptions - README.md: Reorganized as a concise index of available guides, removed redundant content now covered in main README and in-app help - SHUTDOWN_PROCESS.md: Removed (content consolidated into GRACEFUL_SHUTDOWN.md) These updates make the documentation more accurate and easier to navigate for new contributors. --- docs/DEVELOPMENT_WORKFLOW.md | 65 +++++----- docs/README.md | 130 ++++--------------- docs/SHUTDOWN_PROCESS.md | 236 ----------------------------------- 3 files changed, 55 insertions(+), 376 deletions(-) delete mode 100644 docs/SHUTDOWN_PROCESS.md diff --git a/docs/DEVELOPMENT_WORKFLOW.md b/docs/DEVELOPMENT_WORKFLOW.md index f103788..bd2a6b0 100644 --- a/docs/DEVELOPMENT_WORKFLOW.md +++ b/docs/DEVELOPMENT_WORKFLOW.md @@ -16,27 +16,22 @@ This guide covers the development workflow for the open-source Gitea Mirror. 1. **Clone the repository**: ```bash -git clone https://github.com/yourusername/gitea-mirror.git +git clone https://github.com/RayLabsHQ/gitea-mirror.git cd gitea-mirror ``` -2. **Install dependencies**: +2. **Install dependencies and seed the SQLite database**: ```bash -bun install +bun run setup ``` -3. **Initialize database**: -```bash -bun run init-db -``` - -4. **Configure environment**: +3. **Configure environment (optional)**: ```bash cp .env.example .env # Edit .env with your settings ``` -5. **Start development server**: +4. **Start the development server**: ```bash bun run dev ``` @@ -45,29 +40,33 @@ bun run dev | Command | Description | |---------|-------------| -| `bun run dev` | Start development server with hot reload | -| `bun run build` | Build for production | -| `bun run preview` | Preview production build | -| `bun test` | Run all tests | +| `bun run dev` | Start the Bun + Astro dev server with hot reload | +| `bun run build` | Build the production bundle | +| `bun run preview` | Preview the production build locally | +| `bun test` | Run the Bun test suite | | `bun test:watch` | Run tests in watch mode | -| `bun run db:studio` | Open database GUI | +| `bun run db:studio` | Launch Drizzle Kit Studio | ## Project Structure ``` gitea-mirror/ -├── src/ -│ ├── components/ # React components -│ ├── pages/ # Astro pages & API routes -│ ├── lib/ # Core logic -│ │ ├── db/ # Database queries -│ │ ├── utils/ # Helper functions -│ │ └── modules/ # Module system -│ ├── hooks/ # React hooks -│ └── types/ # TypeScript types -├── public/ # Static assets -├── scripts/ # Utility scripts -└── tests/ # Test files +├── src/ # Application UI, API routes, and services +│ ├── components/ # React components rendered inside Astro pages +│ ├── pages/ # Astro pages and API routes (e.g., /api/*) +│ ├── lib/ # Core logic: GitHub/Gitea clients, scheduler, recovery, db helpers +│ │ ├── db/ # Drizzle adapter + schema +│ │ ├── modules/ # Module wiring (jobs, integrations) +│ │ └── utils/ # Shared utilities +│ ├── hooks/ # React hooks +│ ├── content/ # In-app documentation and templated content +│ ├── layouts/ # Shared layout components +│ ├── styles/ # Tailwind CSS entrypoints +│ └── types/ # TypeScript types +├── scripts/ # Bun scripts for DB management and maintenance +├── www/ # Marketing site (Astro + MDX use cases) +├── public/ # Static assets served by Vite/Astro +└── tests/ # Dedicated integration/unit test helpers ``` ## Feature Development @@ -80,10 +79,10 @@ git checkout -b feature/my-feature ``` 2. **Plan your changes**: -- UI components in `/src/components/` -- API endpoints in `/src/pages/api/` -- Database queries in `/src/lib/db/queries/` -- Types in `/src/types/` +- UI components live in `src/components/` +- API endpoints live in `src/pages/api/` +- Database logic is under `src/lib/db/` (schema + adapter) +- Shared types are in `src/types/` 3. **Implement the feature**: @@ -120,7 +119,7 @@ describe('My Feature', () => { 5. **Update documentation**: - Add JSDoc comments -- Update README if needed +- Update README/docs if needed - Document API changes ## Database Development @@ -352,4 +351,4 @@ git push origin v2.23.0 - Check existing [issues](https://github.com/yourusername/gitea-mirror/issues) - Join [discussions](https://github.com/yourusername/gitea-mirror/discussions) -- Read the [FAQ](./FAQ.md) \ No newline at end of file +- Read the [FAQ](./FAQ.md) diff --git a/docs/README.md b/docs/README.md index bd9d272..9b84818 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,123 +1,39 @@ # Gitea Mirror Documentation -Welcome to the Gitea Mirror documentation. This guide covers everything you need to know about developing, building, and deploying the open-source version of Gitea Mirror. +This folder contains engineering and operations references for the open-source Gitea Mirror project. Each guide focuses on the parts of the system that still require bespoke explanation beyond the in-app help and the main `README.md`. -## Documentation Overview +## Available Guides -### Getting Started +### Core workflow +- **[DEVELOPMENT_WORKFLOW.md](./DEVELOPMENT_WORKFLOW.md)** – Set up a local environment, run scripts, and understand the repo layout (app + marketing site). +- **[ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md)** – Complete reference for every configuration flag supported by the app and Docker images. -- **[Development Workflow](./DEVELOPMENT_WORKFLOW.md)** - Set up your development environment and start contributing -- **[Build Guide](./BUILD_GUIDE.md)** - Build Gitea Mirror from source -- **[Configuration Guide](./CONFIGURATION.md)** - Configure all available options +### Reliability & recovery +- **[GRACEFUL_SHUTDOWN.md](./GRACEFUL_SHUTDOWN.md)** – How signal handling, shutdown coordination, and job persistence work in v3. +- **[RECOVERY_IMPROVEMENTS.md](./RECOVERY_IMPROVEMENTS.md)** – Deep dive into the startup recovery workflow and supporting scripts. -### Deployment +### Authentication +- **[SSO-OIDC-SETUP.md](./SSO-OIDC-SETUP.md)** – Configure OIDC/SSO providers through the admin UI. +- **[SSO_TESTING.md](./SSO_TESTING.md)** – Recipes for local and staging SSO testing (Google, Keycloak, mock providers). -- **[Deployment Guide](./DEPLOYMENT.md)** - Deploy to production environments -- **[Docker Guide](./DOCKER.md)** - Container-based deployment -- **[Reverse Proxy Setup](./REVERSE_PROXY.md)** - Configure with nginx/Caddy +If you are looking for customer-facing playbooks, see the MDX use cases under `www/src/pages/use-cases/`. -### Features +## Quick start for local development -- **[SSO/OIDC Setup](./SSO-OIDC-SETUP.md)** - Configure authentication providers -- **[Sponsor Integration](./SPONSOR_INTEGRATION.md)** - GitHub Sponsors integration -- **[Webhook Configuration](./WEBHOOKS.md)** - Set up GitHub webhooks - -### Marketing Site - -- **Use Case Library** (`www/src/pages/use-cases/`) - MDX guides surfaced on the landing page use case section -- **Use Case Cards** (`www/src/lib/use-cases.ts`) - Central place to update titles, summaries, and CTA links shown across the site - -### Architecture - -- **[Architecture Overview](./ARCHITECTURE.md)** - System design and components -- **[API Documentation](./API.md)** - REST API endpoints -- **[Database Schema](./DATABASE.md)** - SQLite structure - -### Maintenance - -- **[Migration Guide](../MIGRATION_GUIDE.md)** - Upgrade from previous versions -- **[Better Auth Migration](./BETTER_AUTH_MIGRATION.md)** - Migrate authentication system -- **[Troubleshooting](./TROUBLESHOOTING.md)** - Common issues and solutions -- **[Backup & Restore](./BACKUP.md)** - Data management - -## Quick Start - -1. **Clone and install**: ```bash -git clone https://github.com/yourusername/gitea-mirror.git +git clone https://github.com/RayLabsHQ/gitea-mirror.git cd gitea-mirror -bun install +bun run setup # installs deps and seeds the SQLite DB +bun run dev # starts the Astro/Bun app on http://localhost:4321 ``` -2. **Configure**: -```bash -cp .env.example .env -# Edit .env with your GitHub and Gitea tokens -``` +The first user you create locally becomes the administrator. All other configuration—GitHub owners, Gitea targets, scheduling, cleanup—is done through the **Configuration** screen in the UI. -3. **Initialize and run**: -```bash -bun run init-db -bun run dev -``` +## Contributing & support -4. **Access**: Open http://localhost:4321 +- 🎯 Contribution guide: [../CONTRIBUTING.md](../CONTRIBUTING.md) +- 📘 Code of conduct: [../CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md) +- 🐞 Issues & feature requests: +- 💬 Discussions: -## Key Features - -- 🔄 **Automatic Syncing** - Keep repositories synchronized -- 🗂️ **Organization Support** - Mirror entire organizations -- ⭐ **Starred Repos** - Mirror your starred repositories -- 🔐 **Self-Hosted** - Full control over your data -- 🚀 **Fast** - Built with Bun for optimal performance -- 🔒 **Secure** - JWT authentication, encrypted tokens - -## Technology Stack - -- **Runtime**: Bun -- **Framework**: Astro with React -- **Database**: SQLite with Drizzle ORM -- **Styling**: Tailwind CSS v4 -- **Authentication**: Better Auth - -## System Requirements - -- Bun >= 1.2.9 -- Node.js >= 20 (optional, for compatibility) -- SQLite 3 -- 512MB RAM minimum -- 1GB disk space - -## Contributing - -We welcome contributions! Please see our [Contributing Guide](../CONTRIBUTING.md) for details. - -### Development Setup - -1. Fork the repository -2. Create a feature branch -3. Make your changes -4. Add tests -5. Submit a pull request - -### Code of Conduct - -Please read our [Code of Conduct](../CODE_OF_CONDUCT.md) before contributing. - -## Support - -- **Issues**: [GitHub Issues](https://github.com/yourusername/gitea-mirror/issues) -- **Discussions**: [GitHub Discussions](https://github.com/yourusername/gitea-mirror/discussions) -- **Wiki**: [GitHub Wiki](https://github.com/yourusername/gitea-mirror/wiki) - -## Security - -For security issues, please see [SECURITY.md](../SECURITY.md). - -## License - -Gitea Mirror is open source software licensed under the [MIT License](../LICENSE). - ---- - -For detailed information on any topic, please refer to the specific documentation guides listed above. +Security disclosures should follow the process in [../SECURITY.md](../SECURITY.md). diff --git a/docs/SHUTDOWN_PROCESS.md b/docs/SHUTDOWN_PROCESS.md deleted file mode 100644 index 4a38625..0000000 --- a/docs/SHUTDOWN_PROCESS.md +++ /dev/null @@ -1,236 +0,0 @@ -# Graceful Shutdown Process - -This document details how the gitea-mirror application handles graceful shutdown during active mirroring operations, with specific focus on job interruption and recovery. - -## Overview - -The graceful shutdown system is designed for **fast, clean termination** without waiting for long-running jobs to complete. It prioritizes **quick shutdown times** (under 30 seconds) while **preserving all progress** for seamless recovery. - -## Key Principle - -**The application does NOT wait for jobs to finish before shutting down.** Instead, it saves the current state and resumes after restart. - -## Shutdown Scenario Example - -### Initial State -- **Job**: Mirror 500 repositories -- **Progress**: 200 repositories completed -- **Remaining**: 300 repositories pending -- **Action**: User initiates shutdown (SIGTERM, Ctrl+C, Docker stop) - -### Shutdown Process (Under 30 seconds) - -#### Step 1: Signal Detection (Immediate) -``` -📡 Received SIGTERM signal -🛑 Graceful shutdown initiated by signal: SIGTERM -📊 Shutdown status: 1 active jobs, 2 callbacks -``` - -#### Step 2: Job State Saving (1-10 seconds) -``` -📝 Step 1: Saving active job states... -Saving state for job abc-123... -✅ Saved state for job abc-123 -``` - -**What gets saved:** -- `inProgress: false` - Mark job as not currently running -- `completedItems: 200` - Number of repos successfully mirrored -- `totalItems: 500` - Total repos in the job -- `completedItemIds: [repo1, repo2, ..., repo200]` - List of completed repos -- `itemIds: [repo1, repo2, ..., repo500]` - Full list of repos -- `lastCheckpoint: 2025-05-24T17:30:00Z` - Exact shutdown time -- `message: "Job interrupted by application shutdown - will resume on restart"` -- `status: "imported"` - Keeps status as resumable (not "failed") - -#### Step 3: Service Cleanup (1-5 seconds) -``` -🔧 Step 2: Executing shutdown callbacks... -🛑 Shutting down cleanup service... -✅ Cleanup service stopped -✅ Shutdown callback 1 completed -``` - -#### Step 4: Clean Exit (Immediate) -``` -💾 Step 3: Closing database connections... -✅ Graceful shutdown completed successfully -``` - -**Total shutdown time: ~15 seconds** (well under the 30-second limit) - -## What Happens to the Remaining 300 Repos? - -### During Shutdown -- **NOT processed** - The remaining 300 repos are not mirrored -- **NOT lost** - Their IDs are preserved in the job state -- **NOT marked as failed** - Job status remains "imported" for recovery - -### After Restart -The recovery system automatically: - -1. **Detects interrupted job** during startup -2. **Calculates remaining work**: 500 - 200 = 300 repos -3. **Extracts remaining repo IDs**: repos 201-500 from the original list -4. **Resumes processing** from exactly where it left off -5. **Continues until completion** of all 500 repos - -## Timeout Configuration - -### Shutdown Timeouts -```typescript -const SHUTDOWN_TIMEOUT = 30000; // 30 seconds max shutdown time -const JOB_SAVE_TIMEOUT = 10000; // 10 seconds to save job state -``` - -### Timeout Behavior -- **Normal case**: Shutdown completes in 10-20 seconds -- **Slow database**: Up to 30 seconds allowed -- **Timeout exceeded**: Force exit with code 1 -- **Container kill**: Orchestrator should allow 45+ seconds grace period - -## Job State Persistence - -### Database Schema -The `mirror_jobs` table stores complete job state: - -```sql --- Job identification -id TEXT PRIMARY KEY, -user_id TEXT NOT NULL, -job_type TEXT NOT NULL DEFAULT 'mirror', - --- Progress tracking -total_items INTEGER, -completed_items INTEGER DEFAULT 0, -item_ids TEXT, -- JSON array of all repo IDs -completed_item_ids TEXT DEFAULT '[]', -- JSON array of completed repo IDs - --- State management -in_progress INTEGER NOT NULL DEFAULT 0, -- Boolean: currently running -started_at TIMESTAMP, -completed_at TIMESTAMP, -last_checkpoint TIMESTAMP, -- Last progress save - --- Status and messaging -status TEXT NOT NULL DEFAULT 'imported', -message TEXT NOT NULL -``` - -### Recovery Query -The recovery system finds interrupted jobs: - -```sql -SELECT * FROM mirror_jobs -WHERE in_progress = 0 - AND status = 'imported' - AND completed_at IS NULL - AND total_items > completed_items; -``` - -## Shutdown-Aware Processing - -### Concurrency Check -During job execution, each repo processing checks for shutdown: - -```typescript -// Before processing each repository -if (isShuttingDown()) { - throw new Error('Processing interrupted by application shutdown'); -} -``` - -### Checkpoint Intervals -Jobs save progress periodically (every 10 repos by default): - -```typescript -checkpointInterval: 10, // Save progress every 10 repositories -``` - -This ensures minimal work loss even if shutdown occurs between checkpoints. - -## Container Integration - -### Docker Entrypoint -The Docker entrypoint properly forwards signals: - -```bash -# Set up signal handlers -trap 'shutdown_handler' TERM INT HUP - -# Start application in background -bun ./dist/server/entry.mjs & -APP_PID=$! - -# Wait for application to finish -wait "$APP_PID" -``` - -### Kubernetes Configuration -Recommended pod configuration: - -```yaml -apiVersion: v1 -kind: Pod -spec: - terminationGracePeriodSeconds: 45 # Allow time for graceful shutdown - containers: - - name: gitea-mirror - # ... other configuration -``` - -## Monitoring and Logging - -### Shutdown Logs -``` -🛑 Graceful shutdown initiated by signal: SIGTERM -📊 Shutdown status: 1 active jobs, 2 callbacks -📝 Step 1: Saving active job states... -Saving state for 1 active jobs... -✅ Completed saving all active jobs -🔧 Step 2: Executing shutdown callbacks... -✅ Completed all shutdown callbacks -💾 Step 3: Closing database connections... -✅ Graceful shutdown completed successfully -``` - -### Recovery Logs -``` -⚠️ Jobs found that need recovery. Starting recovery process... -Resuming job abc-123 with 300 remaining items... -✅ Recovery completed successfully -``` - -## Best Practices - -### For Operations -1. **Monitor shutdown times** - Should complete under 30 seconds -2. **Check recovery logs** - Verify jobs resume correctly after restart -3. **Set appropriate grace periods** - Allow 45+ seconds in orchestrators -4. **Plan maintenance windows** - Jobs will resume but may take time to complete - -### For Development -1. **Test shutdown scenarios** - Use `bun run test-shutdown` -2. **Monitor job progress** - Check checkpoint frequency and timing -3. **Verify recovery** - Ensure interrupted jobs resume correctly -4. **Handle edge cases** - Test shutdown during different job phases - -## Troubleshooting - -### Shutdown Takes Too Long -- **Check**: Database performance during job state saving -- **Solution**: Increase `SHUTDOWN_TIMEOUT` environment variable -- **Monitor**: Job complexity and checkpoint frequency - -### Jobs Don't Resume -- **Check**: Recovery logs for errors during startup -- **Verify**: Database contains interrupted jobs with correct status -- **Test**: Run `bun run startup-recovery` manually - -### Container Force-Killed -- **Check**: Container orchestrator termination grace period -- **Increase**: Grace period to 45+ seconds -- **Monitor**: Application shutdown completion time - -This design ensures **production-ready graceful shutdown** with **zero data loss** and **fast recovery times** suitable for modern containerized deployments.