docs: update development workflow and documentation index

Updated development documentation to reflect current project structure and simplified setup process. Changes: - DEVELOPMENT_WORKFLOW.md: Updated repository URL, simplified setup steps, improved project structure documentation, and clarified command descriptions - README.md: Reorganized as a concise index of available guides, removed redundant content now covered in main README and in-app help - SHUTDOWN_PROCESS.md: Removed (content consolidated into GRACEFUL_SHUTDOWN.md) These updates make the documentation more accurate and easier to navigate for new contributors.
2025-12-06 11:36:44 +03:00 · 2025-10-23 05:10:42 +05:30
parent ea22df1296
commit fd5e68c1d4
3 changed files with 55 additions and 376 deletions
--- a/docs/DEVELOPMENT_WORKFLOW.md
+++ b/docs/DEVELOPMENT_WORKFLOW.md
@@ -16,27 +16,22 @@ This guide covers the development workflow for the open-source Gitea Mirror.

 1. **Clone the repository**:
 ```bash
-git clone https://github.com/yourusername/gitea-mirror.git
+git clone https://github.com/RayLabsHQ/gitea-mirror.git
 cd gitea-mirror
 ```

-2. **Install dependencies**:
+2. **Install dependencies and seed the SQLite database**:
 ```bash
-bun install
+bun run setup
 ```

-3. **Initialize database**:
-```bash
-bun run init-db
-```
-
-4. **Configure environment**:
+3. **Configure environment (optional)**:
 ```bash
 cp .env.example .env
 # Edit .env with your settings
 ```

-5. **Start development server**:
+4. **Start the development server**:
 ```bash
 bun run dev
 ```
@@ -45,29 +40,33 @@ bun run dev

 | Command | Description |
 |---------|-------------|
-| `bun run dev` | Start development server with hot reload |
-| `bun run build` | Build for production |
-| `bun run preview` | Preview production build |
-| `bun test` | Run all tests |
+| `bun run dev` | Start the Bun + Astro dev server with hot reload |
+| `bun run build` | Build the production bundle |
+| `bun run preview` | Preview the production build locally |
+| `bun test` | Run the Bun test suite |
 | `bun test:watch` | Run tests in watch mode |
-| `bun run db:studio` | Open database GUI |
+| `bun run db:studio` | Launch Drizzle Kit Studio |

 ## Project Structure

 ```
 gitea-mirror/
-├── src/
-│   ├── components/      # React components
-│   ├── pages/          # Astro pages & API routes
-│   ├── lib/            # Core logic
-│   │   ├── db/         # Database queries
-│   │   ├── utils/      # Helper functions
-│   │   └── modules/    # Module system
-│   ├── hooks/          # React hooks
-│   └── types/          # TypeScript types
-├── public/             # Static assets
-├── scripts/            # Utility scripts
-└── tests/              # Test files
+├── src/                     # Application UI, API routes, and services
+│   ├── components/          # React components rendered inside Astro pages
+│   ├── pages/               # Astro pages and API routes (e.g., /api/*)
+│   ├── lib/                 # Core logic: GitHub/Gitea clients, scheduler, recovery, db helpers
+│   │   ├── db/              # Drizzle adapter + schema
+│   │   ├── modules/         # Module wiring (jobs, integrations)
+│   │   └── utils/           # Shared utilities
+│   ├── hooks/               # React hooks
+│   ├── content/             # In-app documentation and templated content
+│   ├── layouts/             # Shared layout components
+│   ├── styles/              # Tailwind CSS entrypoints
+│   └── types/               # TypeScript types
+├── scripts/                 # Bun scripts for DB management and maintenance
+├── www/                     # Marketing site (Astro + MDX use cases)
+├── public/                  # Static assets served by Vite/Astro
+└── tests/                   # Dedicated integration/unit test helpers
 ```

 ## Feature Development
@@ -80,10 +79,10 @@ git checkout -b feature/my-feature
 ```

 2. **Plan your changes**:
- UI components in `/src/components/`
- API endpoints in `/src/pages/api/`
- Database queries in `/src/lib/db/queries/`
- Types in `/src/types/`
+- UI components live in `src/components/`
+- API endpoints live in `src/pages/api/`
+- Database logic is under `src/lib/db/` (schema + adapter)
+- Shared types are in `src/types/`

 3. **Implement the feature**:

@@ -120,7 +119,7 @@ describe('My Feature', () => {

 5. **Update documentation**:
 - Add JSDoc comments
- Update README if needed
+- Update README/docs if needed
 - Document API changes

 ## Database Development
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,123 +1,39 @@
 # Gitea Mirror Documentation

-Welcome to the Gitea Mirror documentation. This guide covers everything you need to know about developing, building, and deploying the open-source version of Gitea Mirror.
+This folder contains engineering and operations references for the open-source Gitea Mirror project. Each guide focuses on the parts of the system that still require bespoke explanation beyond the in-app help and the main `README.md`.

-## Documentation Overview
+## Available Guides

-### Getting Started
+### Core workflow
+- **[DEVELOPMENT_WORKFLOW.md](./DEVELOPMENT_WORKFLOW.md)** – Set up a local environment, run scripts, and understand the repo layout (app + marketing site).
+- **[ENVIRONMENT_VARIABLES.md](./ENVIRONMENT_VARIABLES.md)** – Complete reference for every configuration flag supported by the app and Docker images.

- **[Development Workflow](./DEVELOPMENT_WORKFLOW.md)** - Set up your development environment and start contributing
- **[Build Guide](./BUILD_GUIDE.md)** - Build Gitea Mirror from source
- **[Configuration Guide](./CONFIGURATION.md)** - Configure all available options
+### Reliability & recovery
+- **[GRACEFUL_SHUTDOWN.md](./GRACEFUL_SHUTDOWN.md)** – How signal handling, shutdown coordination, and job persistence work in v3.
+- **[RECOVERY_IMPROVEMENTS.md](./RECOVERY_IMPROVEMENTS.md)** – Deep dive into the startup recovery workflow and supporting scripts.

-### Deployment
+### Authentication
+- **[SSO-OIDC-SETUP.md](./SSO-OIDC-SETUP.md)** – Configure OIDC/SSO providers through the admin UI.
+- **[SSO_TESTING.md](./SSO_TESTING.md)** – Recipes for local and staging SSO testing (Google, Keycloak, mock providers).

- **[Deployment Guide](./DEPLOYMENT.md)** - Deploy to production environments
- **[Docker Guide](./DOCKER.md)** - Container-based deployment
- **[Reverse Proxy Setup](./REVERSE_PROXY.md)** - Configure with nginx/Caddy
+If you are looking for customer-facing playbooks, see the MDX use cases under `www/src/pages/use-cases/`.

-### Features
+## Quick start for local development

- **[SSO/OIDC Setup](./SSO-OIDC-SETUP.md)** - Configure authentication providers
- **[Sponsor Integration](./SPONSOR_INTEGRATION.md)** - GitHub Sponsors integration
- **[Webhook Configuration](./WEBHOOKS.md)** - Set up GitHub webhooks
-
-### Marketing Site
-
- **Use Case Library** (`www/src/pages/use-cases/`) - MDX guides surfaced on the landing page use case section
- **Use Case Cards** (`www/src/lib/use-cases.ts`) - Central place to update titles, summaries, and CTA links shown across the site
-
-### Architecture
-
- **[Architecture Overview](./ARCHITECTURE.md)** - System design and components
- **[API Documentation](./API.md)** - REST API endpoints
- **[Database Schema](./DATABASE.md)** - SQLite structure
-
-### Maintenance
-
- **[Migration Guide](../MIGRATION_GUIDE.md)** - Upgrade from previous versions
- **[Better Auth Migration](./BETTER_AUTH_MIGRATION.md)** - Migrate authentication system
- **[Troubleshooting](./TROUBLESHOOTING.md)** - Common issues and solutions
- **[Backup & Restore](./BACKUP.md)** - Data management
-
-## Quick Start
-
-1. **Clone and install**:
 ```bash
-git clone https://github.com/yourusername/gitea-mirror.git
+git clone https://github.com/RayLabsHQ/gitea-mirror.git
 cd gitea-mirror
-bun install
+bun run setup           # installs deps and seeds the SQLite DB
+bun run dev             # starts the Astro/Bun app on http://localhost:4321
 ```

-2. **Configure**:
-```bash
-cp .env.example .env
-# Edit .env with your GitHub and Gitea tokens
-```
+The first user you create locally becomes the administrator. All other configuration—GitHub owners, Gitea targets, scheduling, cleanup—is done through the **Configuration** screen in the UI.

-3. **Initialize and run**:
-```bash
-bun run init-db
-bun run dev
-```
+## Contributing & support

-4. **Access**: Open http://localhost:4321
+- 🎯 Contribution guide: [../CONTRIBUTING.md](../CONTRIBUTING.md)
+- 📘 Code of conduct: [../CODE_OF_CONDUCT.md](../CODE_OF_CONDUCT.md)
+- 🐞 Issues & feature requests: <https://github.com/RayLabsHQ/gitea-mirror/issues>
+- 💬 Discussions: <https://github.com/RayLabsHQ/gitea-mirror/discussions>

-## Key Features
-
- 🔄 **Automatic Syncing** - Keep repositories synchronized
- 🗂️ **Organization Support** - Mirror entire organizations
- ⭐ **Starred Repos** - Mirror your starred repositories
- 🔐 **Self-Hosted** - Full control over your data
- 🚀 **Fast** - Built with Bun for optimal performance
- 🔒 **Secure** - JWT authentication, encrypted tokens
-
-## Technology Stack
-
- **Runtime**: Bun
- **Framework**: Astro with React
- **Database**: SQLite with Drizzle ORM
- **Styling**: Tailwind CSS v4
- **Authentication**: Better Auth
-
-## System Requirements
-
- Bun >= 1.2.9
- Node.js >= 20 (optional, for compatibility)
- SQLite 3
- 512MB RAM minimum
- 1GB disk space
-
-## Contributing
-
-We welcome contributions! Please see our [Contributing Guide](../CONTRIBUTING.md) for details.
-
-### Development Setup
-
-1. Fork the repository
-2. Create a feature branch
-3. Make your changes
-4. Add tests
-5. Submit a pull request
-
-### Code of Conduct
-
-Please read our [Code of Conduct](../CODE_OF_CONDUCT.md) before contributing.
-
-## Support
-
- **Issues**: [GitHub Issues](https://github.com/yourusername/gitea-mirror/issues)
- **Discussions**: [GitHub Discussions](https://github.com/yourusername/gitea-mirror/discussions)
- **Wiki**: [GitHub Wiki](https://github.com/yourusername/gitea-mirror/wiki)
-
-## Security
-
-For security issues, please see [SECURITY.md](../SECURITY.md).
-
-## License
-
-Gitea Mirror is open source software licensed under the [MIT License](../LICENSE).
-
---
-
-For detailed information on any topic, please refer to the specific documentation guides listed above.
+Security disclosures should follow the process in [../SECURITY.md](../SECURITY.md).
--- a/docs/SHUTDOWN_PROCESS.md
+++ b/docs/SHUTDOWN_PROCESS.md
@@ -1,236 +0,0 @@
-# Graceful Shutdown Process
-
-This document details how the gitea-mirror application handles graceful shutdown during active mirroring operations, with specific focus on job interruption and recovery.
-
-## Overview
-
-The graceful shutdown system is designed for **fast, clean termination** without waiting for long-running jobs to complete. It prioritizes **quick shutdown times** (under 30 seconds) while **preserving all progress** for seamless recovery.
-
-## Key Principle
-
-**The application does NOT wait for jobs to finish before shutting down.** Instead, it saves the current state and resumes after restart.
-
-## Shutdown Scenario Example
-
-### Initial State
- **Job**: Mirror 500 repositories
- **Progress**: 200 repositories completed
- **Remaining**: 300 repositories pending
- **Action**: User initiates shutdown (SIGTERM, Ctrl+C, Docker stop)
-
-### Shutdown Process (Under 30 seconds)
-
-#### Step 1: Signal Detection (Immediate)
-```
-📡 Received SIGTERM signal
-🛑 Graceful shutdown initiated by signal: SIGTERM
-📊 Shutdown status: 1 active jobs, 2 callbacks
-```
-
-#### Step 2: Job State Saving (1-10 seconds)
-```
-📝 Step 1: Saving active job states...
-Saving state for job abc-123...
-✅ Saved state for job abc-123
-```
-
-**What gets saved:**
- `inProgress: false` - Mark job as not currently running
- `completedItems: 200` - Number of repos successfully mirrored
- `totalItems: 500` - Total repos in the job
- `completedItemIds: [repo1, repo2, ..., repo200]` - List of completed repos
- `itemIds: [repo1, repo2, ..., repo500]` - Full list of repos
- `lastCheckpoint: 2025-05-24T17:30:00Z` - Exact shutdown time
- `message: "Job interrupted by application shutdown - will resume on restart"`
- `status: "imported"` - Keeps status as resumable (not "failed")
-
-#### Step 3: Service Cleanup (1-5 seconds)
-```
-🔧 Step 2: Executing shutdown callbacks...
-🛑 Shutting down cleanup service...
-✅ Cleanup service stopped
-✅ Shutdown callback 1 completed
-```
-
-#### Step 4: Clean Exit (Immediate)
-```
-💾 Step 3: Closing database connections...
-✅ Graceful shutdown completed successfully
-```
-
-**Total shutdown time: ~15 seconds** (well under the 30-second limit)
-
-## What Happens to the Remaining 300 Repos?
-
-### During Shutdown
- **NOT processed** - The remaining 300 repos are not mirrored
- **NOT lost** - Their IDs are preserved in the job state
- **NOT marked as failed** - Job status remains "imported" for recovery
-
-### After Restart
-The recovery system automatically:
-
-1. **Detects interrupted job** during startup
-2. **Calculates remaining work**: 500 - 200 = 300 repos
-3. **Extracts remaining repo IDs**: repos 201-500 from the original list
-4. **Resumes processing** from exactly where it left off
-5. **Continues until completion** of all 500 repos
-
-## Timeout Configuration
-
-### Shutdown Timeouts
-```typescript
-const SHUTDOWN_TIMEOUT = 30000; // 30 seconds max shutdown time
-const JOB_SAVE_TIMEOUT = 10000; // 10 seconds to save job state
-```
-
-### Timeout Behavior
- **Normal case**: Shutdown completes in 10-20 seconds
- **Slow database**: Up to 30 seconds allowed
- **Timeout exceeded**: Force exit with code 1
- **Container kill**: Orchestrator should allow 45+ seconds grace period
-
-## Job State Persistence
-
-### Database Schema
-The `mirror_jobs` table stores complete job state:
-
-```sql
-- Job identification
-id TEXT PRIMARY KEY,
-user_id TEXT NOT NULL,
-job_type TEXT NOT NULL DEFAULT 'mirror',
-
-- Progress tracking  
-total_items INTEGER,
-completed_items INTEGER DEFAULT 0,
-item_ids TEXT, -- JSON array of all repo IDs
-completed_item_ids TEXT DEFAULT '[]', -- JSON array of completed repo IDs
-
-- State management
-in_progress INTEGER NOT NULL DEFAULT 0, -- Boolean: currently running
-started_at TIMESTAMP,
-completed_at TIMESTAMP,
-last_checkpoint TIMESTAMP, -- Last progress save
-
-- Status and messaging
-status TEXT NOT NULL DEFAULT 'imported',
-message TEXT NOT NULL
-```
-
-### Recovery Query
-The recovery system finds interrupted jobs:
-
-```sql
-SELECT * FROM mirror_jobs 
-WHERE in_progress = 0 
-  AND status = 'imported' 
-  AND completed_at IS NULL
-  AND total_items > completed_items;
-```
-
-## Shutdown-Aware Processing
-
-### Concurrency Check
-During job execution, each repo processing checks for shutdown:
-
-```typescript
-// Before processing each repository
-if (isShuttingDown()) {
-  throw new Error('Processing interrupted by application shutdown');
-}
-```
-
-### Checkpoint Intervals
-Jobs save progress periodically (every 10 repos by default):
-
-```typescript
-checkpointInterval: 10, // Save progress every 10 repositories
-```
-
-This ensures minimal work loss even if shutdown occurs between checkpoints.
-
-## Container Integration
-
-### Docker Entrypoint
-The Docker entrypoint properly forwards signals:
-
-```bash
-# Set up signal handlers
-trap 'shutdown_handler' TERM INT HUP
-
-# Start application in background
-bun ./dist/server/entry.mjs &
-APP_PID=$!
-
-# Wait for application to finish
-wait "$APP_PID"
-```
-
-### Kubernetes Configuration
-Recommended pod configuration:
-
-```yaml
-apiVersion: v1
-kind: Pod
-spec:
-  terminationGracePeriodSeconds: 45  # Allow time for graceful shutdown
-  containers:
-  - name: gitea-mirror
-    # ... other configuration
-```
-
-## Monitoring and Logging
-
-### Shutdown Logs
-```
-🛑 Graceful shutdown initiated by signal: SIGTERM
-📊 Shutdown status: 1 active jobs, 2 callbacks
-📝 Step 1: Saving active job states...
-Saving state for 1 active jobs...
-✅ Completed saving all active jobs
-🔧 Step 2: Executing shutdown callbacks...
-✅ Completed all shutdown callbacks  
-💾 Step 3: Closing database connections...
-✅ Graceful shutdown completed successfully
-```
-
-### Recovery Logs
-```
-⚠️  Jobs found that need recovery. Starting recovery process...
-Resuming job abc-123 with 300 remaining items...
-✅ Recovery completed successfully
-```
-
-## Best Practices
-
-### For Operations
-1. **Monitor shutdown times** - Should complete under 30 seconds
-2. **Check recovery logs** - Verify jobs resume correctly after restart
-3. **Set appropriate grace periods** - Allow 45+ seconds in orchestrators
-4. **Plan maintenance windows** - Jobs will resume but may take time to complete
-
-### For Development
-1. **Test shutdown scenarios** - Use `bun run test-shutdown`
-2. **Monitor job progress** - Check checkpoint frequency and timing
-3. **Verify recovery** - Ensure interrupted jobs resume correctly
-4. **Handle edge cases** - Test shutdown during different job phases
-
-## Troubleshooting
-
-### Shutdown Takes Too Long
- **Check**: Database performance during job state saving
- **Solution**: Increase `SHUTDOWN_TIMEOUT` environment variable
- **Monitor**: Job complexity and checkpoint frequency
-
-### Jobs Don't Resume
- **Check**: Recovery logs for errors during startup
- **Verify**: Database contains interrupted jobs with correct status
- **Test**: Run `bun run startup-recovery` manually
-
-### Container Force-Killed
- **Check**: Container orchestrator termination grace period
- **Increase**: Grace period to 45+ seconds
- **Monitor**: Application shutdown completion time
-
-This design ensures **production-ready graceful shutdown** with **zero data loss** and **fast recovery times** suitable for modern containerized deployments.