🎉 Gitea Mirror: Added

This commit is contained in:
Arunavo Ray
2025-05-18 09:31:23 +05:30
commit 5d40023de0
139 changed files with 22033 additions and 0 deletions

78
scripts/README-docker.md Normal file
View File

@@ -0,0 +1,78 @@
# Scripts Directory
This directory contains utility scripts for the gitea-mirror project.
## Docker Build Script
### build-docker.sh
This script simplifies the process of building and publishing multi-architecture Docker images for the gitea-mirror project.
#### Usage
```bash
./build-docker.sh [--load] [--push]
```
Options:
- `--load`: Load the built image into the local Docker daemon
- `--push`: Push the image to the configured Docker registry
Without any flags, the script will build the image but leave it in the build cache only.
#### Configuration
The script uses environment variables from the `.env` file in the project root:
- `DOCKER_REGISTRY`: The Docker registry to push to (default: ghcr.io)
- `DOCKER_IMAGE`: The image name (default: gitea-mirror)
- `DOCKER_TAG`: The image tag (default: latest)
#### Examples
1. Build for multiple architectures and load into Docker:
```bash
./scripts/build-docker.sh --load
```
2. Build and push to the registry:
```bash
./scripts/build-docker.sh --push
```
3. Using with docker-compose:
```bash
# Ensure dependencies are installed and database is initialized
pnpm setup
# First build the image
./scripts/build-docker.sh --load
# Then run using docker-compose for development
docker-compose -f ../docker-compose.dev.yml up -d
# Or for production
docker-compose --profile production up -d
```
## Diagnostics Script
### docker-diagnostics.sh
This utility script helps diagnose issues with your Docker setup for building and running Gitea Mirror.
#### Usage
```bash
./scripts/docker-diagnostics.sh
```
The script checks:
- Docker and Docker Compose installation
- Docker Buildx configuration
- QEMU availability for multi-architecture builds
- Docker resources (memory, CPU)
- Environment configuration
- Provides recommendations for building and troubleshooting
Run this script before building if you're experiencing issues with Docker builds or want to validate your environment.

58
scripts/README.md Normal file
View File

@@ -0,0 +1,58 @@
# Scripts Directory
This folder contains utility scripts for database management.
## Database Management Tool (manage-db.ts)
This is a consolidated database management tool that handles all database-related operations. It combines the functionality of the previous separate scripts into a single, more intelligent script that can check, fix, and initialize the database as needed.
### Features
- **Check Mode**: Validates the existence and integrity of the database
- **Init Mode**: Creates the database only if it doesn't already exist
- **Fix Mode**: Corrects database file location issues
- **Reset Users Mode**: Removes all users and their data
- **Auto Mode**: Automatically checks, fixes, and initializes the database if needed
## Running the Database Management Tool
You can execute the database management tool using your package manager with various commands:
```bash
# Checks database status (default action if no command is specified, equivalent to 'pnpm check-db')
pnpm manage-db
# Check database status
pnpm check-db
# Initialize the database (only if it doesn't exist)
pnpm init-db
# Fix database location issues
pnpm fix-db
# Automatic check, fix, and initialize if needed
pnpm db-auto
# Reset all users (for testing signup flow)
pnpm reset-users
# Update the database schema to the latest version
pnpm update-schema
# Remove database files completely
pnpm cleanup-db
# Complete setup (install dependencies and initialize database)
pnpm setup
# Start development server with a fresh database
pnpm dev:clean
# Start production server with a fresh database
pnpm start:fresh
```
## Database File Location
The database file should be located in the `./data/gitea-mirror.db` directory. If the file is found in the root directory, the fix mode will move it to the correct location.

104
scripts/build-docker.sh Executable file
View File

@@ -0,0 +1,104 @@
#!/bin/bash
# Build and push the Gitea Mirror docker image for multiple architectures
set -e # Exit on any error
# Load environment variables if .env file exists
if [ -f .env ]; then
echo "Loading environment variables from .env"
export $(grep -v '^#' .env | xargs)
fi
# Set default values if not set in environment
DOCKER_REGISTRY=${DOCKER_REGISTRY:-ghcr.io}
DOCKER_IMAGE=${DOCKER_IMAGE:-gitea-mirror}
DOCKER_TAG=${DOCKER_TAG:-latest}
FULL_IMAGE_NAME="$DOCKER_REGISTRY/$DOCKER_IMAGE:$DOCKER_TAG"
echo "Building image: $FULL_IMAGE_NAME"
# Parse command line arguments
LOAD=false
PUSH=false
while [[ $# -gt 0 ]]; do
key="$1"
case $key in
--load)
LOAD=true
shift
;;
--push)
PUSH=true
shift
;;
*)
echo "Unknown option: $key"
echo "Usage: $0 [--load] [--push]"
echo " --load Load the image into Docker after build"
echo " --push Push the image to the registry after build"
exit 1
;;
esac
done
# Build command construction
BUILD_CMD="docker buildx build --platform linux/amd64,linux/arm64 -t $FULL_IMAGE_NAME"
# Add load or push flag if specified
if [ "$LOAD" = true ]; then
BUILD_CMD="$BUILD_CMD --load"
fi
if [ "$PUSH" = true ]; then
BUILD_CMD="$BUILD_CMD --push"
fi
# Add context directory
BUILD_CMD="$BUILD_CMD ."
# Execute the build command
echo "Executing: $BUILD_CMD"
# Function to execute with retries
execute_with_retry() {
local cmd="$1"
local max_attempts=${2:-3}
local attempt=1
local delay=5
while [ $attempt -le $max_attempts ]; do
echo "Attempt $attempt of $max_attempts..."
if eval "$cmd"; then
echo "Command succeeded!"
return 0
else
echo "Command failed, waiting $delay seconds before retry..."
sleep $delay
attempt=$((attempt + 1))
delay=$((delay * 2)) # Exponential backoff
fi
done
echo "All attempts failed!"
return 1
}
# Execute with retry
execute_with_retry "$BUILD_CMD"
BUILD_RESULT=$?
if [ $BUILD_RESULT -eq 0 ]; then
echo "✅ Build successful!"
else
echo "❌ Build failed after multiple attempts."
exit 1
fi
# Print help message if neither --load nor --push was specified
if [ "$LOAD" = false ] && [ "$PUSH" = false ]; then
echo
echo "NOTE: Image was built but not loaded or pushed. To use this image, run again with:"
echo " $0 --load # to load into local Docker"
echo " $0 --push # to push to registry $DOCKER_REGISTRY"
fi

125
scripts/docker-diagnostics.sh Executable file
View File

@@ -0,0 +1,125 @@
#!/bin/bash
# Docker setup diagnostics tool for Gitea Mirror
# ANSI color codes
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${BLUE}=====================================================${NC}"
echo -e "${BLUE} Gitea Mirror Docker Setup Diagnostics ${NC}"
echo -e "${BLUE}=====================================================${NC}"
# Check if Docker is installed and running
echo -e "\n${YELLOW}Checking Docker...${NC}"
if command -v docker &> /dev/null; then
echo -e "${GREEN}✓ Docker is installed${NC}"
if docker info &> /dev/null; then
echo -e "${GREEN}✓ Docker daemon is running${NC}"
# Get Docker version
DOCKER_VERSION=$(docker version --format '{{.Server.Version}}')
echo -e "${GREEN}✓ Docker version: $DOCKER_VERSION${NC}"
else
echo -e "${RED}✗ Docker daemon is not running${NC}"
echo -e " Run: ${YELLOW}open -a Docker${NC}"
fi
else
echo -e "${RED}✗ Docker is not installed${NC}"
echo -e " Visit: ${BLUE}https://www.docker.com/products/docker-desktop${NC}"
fi
# Check for Docker Compose
echo -e "\n${YELLOW}Checking Docker Compose...${NC}"
if docker compose version &> /dev/null; then
COMPOSE_VERSION=$(docker compose version --short)
echo -e "${GREEN}✓ Docker Compose is installed (v$COMPOSE_VERSION)${NC}"
elif command -v docker-compose &> /dev/null; then
COMPOSE_VERSION=$(docker-compose --version | awk '{print $3}' | sed 's/,//')
echo -e "${GREEN}✓ Docker Compose is installed (v$COMPOSE_VERSION)${NC}"
echo -e "${YELLOW}⚠ Using legacy docker-compose - consider upgrading${NC}"
else
echo -e "${RED}✗ Docker Compose is not installed${NC}"
fi
# Check for Docker Buildx
echo -e "\n${YELLOW}Checking Docker Buildx...${NC}"
if docker buildx version &> /dev/null; then
BUILDX_VERSION=$(docker buildx version | head -n1 | awk '{print $2}')
echo -e "${GREEN}✓ Docker Buildx is installed (v$BUILDX_VERSION)${NC}"
# List available builders
echo -e "\n${YELLOW}Available builders:${NC}"
docker buildx ls
else
echo -e "${RED}✗ Docker Buildx is not installed or not activated${NC}"
fi
# Check for QEMU
echo -e "\n${YELLOW}Checking QEMU for multi-platform builds...${NC}"
if docker run --rm --privileged multiarch/qemu-user-static --reset -p yes &> /dev/null; then
echo -e "${GREEN}✓ QEMU is available for multi-architecture builds${NC}"
else
echo -e "${RED}✗ QEMU setup issue - multi-platform builds may fail${NC}"
echo -e " Run: ${YELLOW}docker run --rm --privileged multiarch/qemu-user-static --reset -p yes${NC}"
fi
# Check Docker resources
echo -e "\n${YELLOW}Checking Docker resources...${NC}"
if [ "$(uname)" == "Darwin" ]; then
# macOS
if command -v osascript &> /dev/null; then
SYS_MEM=$(( $(sysctl -n hw.memsize) / 1024 / 1024 / 1024 ))
echo -e "System memory: ${GREEN}$SYS_MEM GB${NC}"
echo -e "NOTE: Check Docker Desktop settings to see allocated resources"
echo -e "Recommended: At least 4GB RAM and 2 CPUs for multi-platform builds"
fi
fi
# Check environment file
echo -e "\n${YELLOW}Checking environment configuration...${NC}"
if [ -f .env ]; then
echo -e "${GREEN}✓ .env file exists${NC}"
# Parse .env file safely
if [ -f .env ]; then
REGISTRY=$(grep DOCKER_REGISTRY .env | cut -d= -f2)
IMAGE=$(grep DOCKER_IMAGE .env | cut -d= -f2)
TAG=$(grep DOCKER_TAG .env | cut -d= -f2)
echo -e "Docker image configuration:"
echo -e " Registry: ${BLUE}${REGISTRY:-"Not set (will use default)"}${NC}"
echo -e " Image: ${BLUE}${IMAGE:-"Not set (will use default)"}${NC}"
echo -e " Tag: ${BLUE}${TAG:-"Not set (will use default)"}${NC}"
fi
else
echo -e "${YELLOW}⚠ .env file not found${NC}"
echo -e " Run: ${YELLOW}cp .env.example .env${NC}"
fi
# Conclusion and recommendations
echo -e "\n${BLUE}=====================================================${NC}"
echo -e "${BLUE} Recommendations ${NC}"
echo -e "${BLUE}=====================================================${NC}"
echo -e "\n${YELLOW}For local development:${NC}"
echo -e "1. ${GREEN}pnpm setup${NC} (initialize database and install dependencies)"
echo -e "2. ${GREEN}./scripts/build-docker.sh --load${NC} (build and load into Docker)"
echo -e "3. ${GREEN}docker-compose -f docker-compose.dev.yml up -d${NC} (start the development container)"
echo -e "\n${YELLOW}For production deployment (using Docker Compose):${NC}"
echo -e "1. ${GREEN}pnpm setup${NC} (if not already done, to ensure database schema is ready)"
echo -e "2. ${GREEN}docker-compose --profile production up -d${NC} (start the production container)"
echo -e "\n${YELLOW}For CI/CD builds:${NC}"
echo -e "1. Use GitHub Actions workflow with retry mechanism"
echo -e "2. If build fails, try running with: ${GREEN}DOCKER_BUILDKIT=1${NC}"
echo -e "3. Consider breaking the build into multiple steps for better reliability"
echo -e "\n${YELLOW}For troubleshooting:${NC}"
echo -e "1. Check container logs: ${GREEN}docker logs gitea-mirror-dev${NC} (for development) or ${GREEN}docker logs gitea-mirror${NC} (for production)"
echo -e "2. Check health status: ${GREEN}docker inspect --format='{{.State.Health.Status}}' gitea-mirror-dev${NC} (for development) or ${GREEN}docker inspect --format='{{.State.Health.Status}}' gitea-mirror${NC} (for production)"
echo -e "3. See full documentation: ${BLUE}.github/workflows/TROUBLESHOOTING.md${NC}"
echo -e ""

803
scripts/manage-db.ts Normal file
View File

@@ -0,0 +1,803 @@
import fs from "fs";
import path from "path";
import { client, db } from "../src/lib/db";
import { configs } from "../src/lib/db";
import { v4 as uuidv4 } from "uuid";
// Command line arguments
const args = process.argv.slice(2);
const command = args[0] || "check";
// Ensure data directory exists
const dataDir = path.join(process.cwd(), "data");
if (!fs.existsSync(dataDir)) {
fs.mkdirSync(dataDir, { recursive: true });
}
// Database paths
const rootDbFile = path.join(process.cwd(), "gitea-mirror.db");
const rootDevDbFile = path.join(process.cwd(), "gitea-mirror-dev.db");
const dataDbFile = path.join(dataDir, "gitea-mirror.db");
const dataDevDbFile = path.join(dataDir, "gitea-mirror-dev.db");
// Database path - ensure we use absolute path
const dbPath =
process.env.DATABASE_URL || `file:${path.join(dataDir, "gitea-mirror.db")}`;
/**
* Ensure all required tables exist
*/
async function ensureTablesExist() {
const requiredTables = [
"users",
"configs",
"repositories",
"organizations",
"mirror_jobs",
];
for (const table of requiredTables) {
try {
await client.execute(`SELECT 1 FROM ${table} LIMIT 1`);
} catch (error) {
if (error instanceof Error && error.message.includes("SQLITE_ERROR")) {
console.warn(`⚠️ Table '${table}' is missing. Creating it now...`);
switch (table) {
case "users":
await client.execute(
`CREATE TABLE users (
id TEXT PRIMARY KEY,
username TEXT NOT NULL,
password TEXT NOT NULL,
email TEXT NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)`
);
break;
case "configs":
await client.execute(
`CREATE TABLE configs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
github_config TEXT NOT NULL,
gitea_config TEXT NOT NULL,
include TEXT NOT NULL DEFAULT '[]',
exclude TEXT NOT NULL DEFAULT '[]',
schedule_config TEXT NOT NULL,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id)
)`
);
break;
case "repositories":
await client.execute(
`CREATE TABLE repositories (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
full_name TEXT NOT NULL,
url TEXT NOT NULL,
clone_url TEXT NOT NULL,
owner TEXT NOT NULL,
organization TEXT,
is_private INTEGER NOT NULL DEFAULT 0,
is_fork INTEGER NOT NULL DEFAULT 0,
forked_from TEXT,
has_issues INTEGER NOT NULL DEFAULT 0,
is_starred INTEGER NOT NULL DEFAULT 0,
is_archived INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
has_lfs INTEGER NOT NULL DEFAULT 0,
has_submodules INTEGER NOT NULL DEFAULT 0,
default_branch TEXT NOT NULL,
visibility TEXT NOT NULL DEFAULT 'public',
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
)`
);
break;
case "organizations":
await client.execute(
`CREATE TABLE organizations (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT NOT NULL,
membership_role TEXT NOT NULL DEFAULT 'member',
is_included INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
repository_count INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
)`
);
break;
case "mirror_jobs":
await client.execute(
`CREATE TABLE mirror_jobs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
repository_id TEXT,
repository_name TEXT,
organization_id TEXT,
organization_name TEXT,
details TEXT,
status TEXT NOT NULL DEFAULT 'imported',
message TEXT NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (user_id) REFERENCES users(id)
)`
);
break;
}
console.log(`✅ Table '${table}' created successfully.`);
} else {
console.error(`❌ Error checking table '${table}':`, error);
process.exit(1);
}
}
}
}
/**
* Check database status
*/
async function checkDatabase() {
console.log("Checking database status...");
// Check for database files in the root directory (which is incorrect)
if (fs.existsSync(rootDbFile)) {
console.warn(
"⚠️ WARNING: Database file found in root directory: gitea-mirror.db"
);
console.warn(" This file should be in the data directory.");
console.warn(
' Run "pnpm manage-db fix" to fix this issue or "pnpm cleanup-db" to remove it.'
);
}
// Check if database files exist in the data directory (which is correct)
if (fs.existsSync(dataDbFile)) {
console.log(
"✅ Database file found in data directory: data/gitea-mirror.db"
);
// Check for users
try {
const userCountResult = await client.execute(
`SELECT COUNT(*) as count FROM users`
);
const userCount = userCountResult.rows[0].count;
if (userCount === 0) {
console.log(" No users found in the database.");
console.log(
" When you start the application, you will be directed to the signup page"
);
console.log(" to create an initial admin account.");
} else {
console.log(`${userCount} user(s) found in the database.`);
console.log(" The application will show the login page on startup.");
}
// Check for configurations
const configCountResult = await client.execute(
`SELECT COUNT(*) as count FROM configs`
);
const configCount = configCountResult.rows[0].count;
if (configCount === 0) {
console.log(" No configurations found in the database.");
console.log(
" You will need to set up your GitHub and Gitea configurations after login."
);
} else {
console.log(
`${configCount} configuration(s) found in the database.`
);
}
} catch (error) {
console.error("❌ Error connecting to the database:", error);
console.warn(
' The database file might be corrupted. Consider running "pnpm manage-db init" to recreate it.'
);
}
} else {
console.warn("⚠️ WARNING: Database file not found in data directory.");
console.warn(' Run "pnpm manage-db init" to create it.');
}
}
/**
* Update database schema
*/
async function updateSchema() {
console.log(`Checking and updating database schema at ${dbPath}...`);
// Check if the database exists
if (!fs.existsSync(dataDbFile)) {
console.log(
"⚠️ Database file doesn't exist. Run 'pnpm manage-db init' first to create it."
);
return;
}
try {
console.log("Checking for missing columns in mirror_jobs table...");
// Check if repository_id column exists in mirror_jobs table
const tableInfoResult = await client.execute(
`PRAGMA table_info(mirror_jobs)`
);
// Get column names
const columns = tableInfoResult.rows.map((row) => row.name);
// Check for repository_id column
if (!columns.includes("repository_id")) {
console.log(
"Adding missing repository_id column to mirror_jobs table..."
);
await client.execute(
`ALTER TABLE mirror_jobs ADD COLUMN repository_id TEXT;`
);
console.log("✅ Added repository_id column to mirror_jobs table.");
}
// Check for repository_name column
if (!columns.includes("repository_name")) {
console.log(
"Adding missing repository_name column to mirror_jobs table..."
);
await client.execute(
`ALTER TABLE mirror_jobs ADD COLUMN repository_name TEXT;`
);
console.log("✅ Added repository_name column to mirror_jobs table.");
}
// Check for organization_id column
if (!columns.includes("organization_id")) {
console.log(
"Adding missing organization_id column to mirror_jobs table..."
);
await client.execute(
`ALTER TABLE mirror_jobs ADD COLUMN organization_id TEXT;`
);
console.log("✅ Added organization_id column to mirror_jobs table.");
}
// Check for organization_name column
if (!columns.includes("organization_name")) {
console.log(
"Adding missing organization_name column to mirror_jobs table..."
);
await client.execute(
`ALTER TABLE mirror_jobs ADD COLUMN organization_name TEXT;`
);
console.log("✅ Added organization_name column to mirror_jobs table.");
}
// Check for details column
if (!columns.includes("details")) {
console.log("Adding missing details column to mirror_jobs table...");
await client.execute(`ALTER TABLE mirror_jobs ADD COLUMN details TEXT;`);
console.log("✅ Added details column to mirror_jobs table.");
}
// Check for mirrored_location column in repositories table
const repoColumns = await client.execute(
`PRAGMA table_info(repositories)`
);
const repoColumnNames = repoColumns.rows.map((row: any) => row.name);
if (!repoColumnNames.includes("mirrored_location")) {
console.log("Adding missing mirrored_location column to repositories table...");
await client.execute(
`ALTER TABLE repositories ADD COLUMN mirrored_location TEXT DEFAULT '';`
);
console.log("✅ Added mirrored_location column to repositories table.");
}
console.log("✅ Schema update completed successfully.");
} catch (error) {
console.error("❌ Error updating schema:", error);
process.exit(1);
}
}
/**
* Initialize the database
*/
async function initializeDatabase() {
// Check if database already exists first
if (fs.existsSync(dataDbFile)) {
console.log("⚠️ Database already exists at data/gitea-mirror.db");
console.log(
' If you want to recreate the database, run "pnpm cleanup-db" first.'
);
console.log(
' Or use "pnpm manage-db reset-users" to just remove users without recreating tables.'
);
// Check if we can connect to it
try {
await client.execute(`SELECT COUNT(*) as count FROM users`);
console.log("✅ Database is valid and accessible.");
return;
} catch (error) {
console.error("❌ Error connecting to the existing database:", error);
console.log(
" The database might be corrupted. Proceeding with reinitialization..."
);
}
}
console.log(`Initializing database at ${dbPath}...`);
try {
// Create tables if they don't exist
await client.execute(
`CREATE TABLE IF NOT EXISTS users (
id TEXT PRIMARY KEY,
username TEXT NOT NULL,
password TEXT NOT NULL,
email TEXT NOT NULL,
created_at INTEGER NOT NULL,
updated_at INTEGER NOT NULL
)`
);
// NOTE: We no longer create a default admin user - user will create one via signup page
await client.execute(
`CREATE TABLE IF NOT EXISTS configs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
name TEXT NOT NULL,
is_active INTEGER NOT NULL DEFAULT 1,
github_config TEXT NOT NULL,
gitea_config TEXT NOT NULL,
include TEXT NOT NULL DEFAULT '["*"]',
exclude TEXT NOT NULL DEFAULT '[]',
schedule_config TEXT NOT NULL,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id)
);
`
);
await client.execute(
`CREATE TABLE IF NOT EXISTS repositories (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
full_name TEXT NOT NULL,
url TEXT NOT NULL,
clone_url TEXT NOT NULL,
owner TEXT NOT NULL,
organization TEXT,
mirrored_location TEXT DEFAULT '',
is_private INTEGER NOT NULL DEFAULT 0,
is_fork INTEGER NOT NULL DEFAULT 0,
forked_from TEXT,
has_issues INTEGER NOT NULL DEFAULT 0,
is_starred INTEGER NOT NULL DEFAULT 0,
is_archived INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0,
has_lfs INTEGER NOT NULL DEFAULT 0,
has_submodules INTEGER NOT NULL DEFAULT 0,
default_branch TEXT NOT NULL,
visibility TEXT NOT NULL DEFAULT 'public',
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
);
`
);
await client.execute(
`CREATE TABLE IF NOT EXISTS organizations (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
config_id TEXT NOT NULL,
name TEXT NOT NULL,
avatar_url TEXT NOT NULL,
membership_role TEXT NOT NULL DEFAULT 'member',
is_included INTEGER NOT NULL DEFAULT 1,
status TEXT NOT NULL DEFAULT 'imported',
last_mirrored INTEGER,
error_message TEXT,
repository_count INTEGER NOT NULL DEFAULT 0,
created_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
updated_at INTEGER NOT NULL DEFAULT (strftime('%s','now')),
FOREIGN KEY (user_id) REFERENCES users(id),
FOREIGN KEY (config_id) REFERENCES configs(id)
);
`
);
await client.execute(
`CREATE TABLE IF NOT EXISTS mirror_jobs (
id TEXT PRIMARY KEY,
user_id TEXT NOT NULL,
repository_id TEXT,
repository_name TEXT,
organization_id TEXT,
organization_name TEXT,
details TEXT,
status TEXT NOT NULL DEFAULT 'imported',
message TEXT NOT NULL,
timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (user_id) REFERENCES users(id)
);
`
);
// Insert default config if none exists
const configCountResult = await client.execute(
`SELECT COUNT(*) as count FROM configs`
);
const configCount = configCountResult.rows[0].count;
if (configCount === 0) {
// Get the first user
const firstUserResult = await client.execute(
`SELECT id FROM users LIMIT 1`
);
if (firstUserResult.rows.length > 0) {
const userId = firstUserResult.rows[0].id;
const configId = uuidv4();
const githubConfig = JSON.stringify({
username: process.env.GITHUB_USERNAME || "",
token: process.env.GITHUB_TOKEN || "",
skipForks: false,
privateRepositories: false,
mirrorIssues: false,
mirrorStarred: true,
useSpecificUser: false,
preserveOrgStructure: true,
skipStarredIssues: false,
});
const giteaConfig = JSON.stringify({
url: process.env.GITEA_URL || "",
token: process.env.GITEA_TOKEN || "",
username: process.env.GITEA_USERNAME || "",
organization: "",
visibility: "public",
starredReposOrg: "github",
});
const include = JSON.stringify(["*"]);
const exclude = JSON.stringify([]);
const scheduleConfig = JSON.stringify({
enabled: false,
interval: 3600,
lastRun: null,
nextRun: null,
});
await client.execute(
`
INSERT INTO configs (id, user_id, name, is_active, github_config, gitea_config, include, exclude, schedule_config, created_at, updated_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
`,
[
configId,
userId,
"Default Configuration",
1,
githubConfig,
giteaConfig,
include,
exclude,
scheduleConfig,
Date.now(),
Date.now(),
]
);
}
}
console.log("✅ Database initialization completed successfully.");
} catch (error) {
console.error("❌ Error initializing database:", error);
process.exit(1);
}
}
/**
* Reset users in the database
*/
async function resetUsers() {
console.log(`Resetting users in database at ${dbPath}...`);
try {
// Check if the database exists
const dbFilePath = dbPath.replace("file:", "");
const doesDbExist = fs.existsSync(dbFilePath);
if (!doesDbExist) {
console.log(
"❌ Database file doesn't exist. Run 'pnpm manage-db init' first to create it."
);
return;
}
// Count existing users
const userCountResult = await client.execute(
`SELECT COUNT(*) as count FROM users`
);
const userCount = userCountResult.rows[0].count;
if (userCount === 0) {
console.log(" No users found in the database. Nothing to reset.");
return;
}
// Delete all users
await client.execute(`DELETE FROM users`);
console.log(`✅ Deleted ${userCount} users from the database.`);
// Check dependent configurations that need to be removed
const configCount = await client.execute(
`SELECT COUNT(*) as count FROM configs`
);
if (
configCount.rows &&
configCount.rows[0] &&
Number(configCount.rows[0].count) > 0
) {
await client.execute(`DELETE FROM configs`);
console.log(`✅ Deleted ${configCount.rows[0].count} configurations.`);
}
// Check for dependent repositories
const repoCount = await client.execute(
`SELECT COUNT(*) as count FROM repositories`
);
if (
repoCount.rows &&
repoCount.rows[0] &&
Number(repoCount.rows[0].count) > 0
) {
await client.execute(`DELETE FROM repositories`);
console.log(`✅ Deleted ${repoCount.rows[0].count} repositories.`);
}
// Check for dependent organizations
const orgCount = await client.execute(
`SELECT COUNT(*) as count FROM organizations`
);
if (
orgCount.rows &&
orgCount.rows[0] &&
Number(orgCount.rows[0].count) > 0
) {
await client.execute(`DELETE FROM organizations`);
console.log(`✅ Deleted ${orgCount.rows[0].count} organizations.`);
}
// Check for dependent mirror jobs
const jobCount = await client.execute(
`SELECT COUNT(*) as count FROM mirror_jobs`
);
if (
jobCount.rows &&
jobCount.rows[0] &&
Number(jobCount.rows[0].count) > 0
) {
await client.execute(`DELETE FROM mirror_jobs`);
console.log(`✅ Deleted ${jobCount.rows[0].count} mirror jobs.`);
}
console.log(
"✅ Database has been reset. The application will now prompt for a new admin account setup on next run."
);
} catch (error) {
console.error("❌ Error resetting users:", error);
process.exit(1);
}
}
/**
* Fix database location issues
*/
async function fixDatabaseIssues() {
console.log("Checking for database issues...");
// Check for database files in the root directory
if (fs.existsSync(rootDbFile)) {
console.log("Found database file in root directory: gitea-mirror.db");
// If the data directory doesn't have the file, move it there
if (!fs.existsSync(dataDbFile)) {
console.log("Moving database file to data directory...");
fs.copyFileSync(rootDbFile, dataDbFile);
console.log("Database file moved successfully.");
} else {
console.log(
"Database file already exists in data directory. Checking for differences..."
);
// Compare file sizes to see which is newer/larger
const rootStats = fs.statSync(rootDbFile);
const dataStats = fs.statSync(dataDbFile);
if (
rootStats.size > dataStats.size ||
rootStats.mtime > dataStats.mtime
) {
console.log(
"Root database file is newer or larger. Backing up data directory file and replacing it..."
);
fs.copyFileSync(dataDbFile, `${dataDbFile}.backup-${Date.now()}`);
fs.copyFileSync(rootDbFile, dataDbFile);
console.log("Database file replaced successfully.");
}
}
// Remove the root file
console.log("Removing database file from root directory...");
fs.unlinkSync(rootDbFile);
console.log("Root database file removed.");
}
// Do the same for dev database
if (fs.existsSync(rootDevDbFile)) {
console.log(
"Found development database file in root directory: gitea-mirror-dev.db"
);
// If the data directory doesn't have the file, move it there
if (!fs.existsSync(dataDevDbFile)) {
console.log("Moving development database file to data directory...");
fs.copyFileSync(rootDevDbFile, dataDevDbFile);
console.log("Development database file moved successfully.");
} else {
console.log(
"Development database file already exists in data directory. Checking for differences..."
);
// Compare file sizes to see which is newer/larger
const rootStats = fs.statSync(rootDevDbFile);
const dataStats = fs.statSync(dataDevDbFile);
if (
rootStats.size > dataStats.size ||
rootStats.mtime > dataStats.mtime
) {
console.log(
"Root development database file is newer or larger. Backing up data directory file and replacing it..."
);
fs.copyFileSync(dataDevDbFile, `${dataDevDbFile}.backup-${Date.now()}`);
fs.copyFileSync(rootDevDbFile, dataDevDbFile);
console.log("Development database file replaced successfully.");
}
}
// Remove the root file
console.log("Removing development database file from root directory...");
fs.unlinkSync(rootDevDbFile);
console.log("Root development database file removed.");
}
// Check if database files exist in the data directory
if (!fs.existsSync(dataDbFile)) {
console.warn(
"⚠️ WARNING: Production database file not found in data directory."
);
console.warn(' Run "pnpm manage-db init" to create it.');
} else {
console.log("✅ Production database file found in data directory.");
// Check if we can connect to the database
try {
// Try to query the database
const configCount = await db.select().from(configs).limit(1);
console.log(`✅ Successfully connected to the database.`);
} catch (error) {
console.error("❌ Error connecting to the database:", error);
console.warn(
' The database file might be corrupted. Consider running "pnpm manage-db init" to recreate it.'
);
}
}
console.log("Database check completed.");
}
/**
* Main function to handle the command
*/
async function main() {
console.log(`Database Management Tool for Gitea Mirror`);
// Ensure all required tables exist
console.log("Ensuring all required tables exist...");
await ensureTablesExist();
switch (command) {
case "check":
await checkDatabase();
break;
case "init":
await initializeDatabase();
break;
case "fix":
await fixDatabaseIssues();
break;
case "reset-users":
await resetUsers();
break;
case "update-schema":
await updateSchema();
break;
case "auto":
// Auto mode: check, fix, and initialize if needed
console.log("Running in auto mode: check, fix, and initialize if needed");
await fixDatabaseIssues();
// Also update schema in auto mode
await updateSchema();
if (!fs.existsSync(dataDbFile)) {
await initializeDatabase();
} else {
await checkDatabase();
}
break;
default:
console.log(`
Available commands:
check - Check database status
init - Initialize the database (only if it doesn't exist)
fix - Fix database location issues
reset-users - Remove all users and their data
update-schema - Update the database schema to the latest version
auto - Automatic mode: check, fix, and initialize if needed
Usage: pnpm manage-db [command]
`);
}
}
main().catch((error) => {
console.error("Error during database management:", error);
process.exit(1);
});

18
scripts/run-migrations.ts Normal file
View File

@@ -0,0 +1,18 @@
import { addMirroredLocationColumn } from "../src/lib/db/migrations/add-mirrored-location";
async function runMigrations() {
try {
console.log("Running database migrations...");
// Run the migration to add the mirrored_location column
await addMirroredLocationColumn();
console.log("All migrations completed successfully");
process.exit(0);
} catch (error) {
console.error("Migration failed:", error);
process.exit(1);
}
}
runMigrations();