Mirror/gitea-mirror
1
0
Fork 0
mirror of https://github.com/RayLabsHQ/gitea-mirror.git synced 2025-12-06 19:46:44 +03:00
Code Issues Packages Projects Releases Wiki Activity

Remove Auto Migrate

Browse Source
This commit is contained in:
Arunavo Ray
2025-07-19 00:28:12 +05:30
parent 67080a7ce9
commit 1e06e2bd4b
7 changed files with 77 additions and 402 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@@ -11,7 +11,7 @@
</p>
> [!IMPORTANT]
> **Upgrading to v3?** Please read the [Migration Guide](MIGRATION_GUIDE.md) for breaking changes and upgrade instructions.
> **Upgrading to v3?** v3 requires a fresh start with a new data volume. Please read the [Upgrade Guide](UPGRADE.md) for instructions.
## 🚀 Quick Start
@@ -35,7 +35,7 @@ First user signup becomes admin. Configure GitHub and Gitea through the web inte
- 🔁 Mirror public, private, and starred GitHub repos to Gitea
- 🏢 Mirror entire organizations with flexible strategies
- 🎯 Custom destination control for repos and organizations
- 🔐 Secure authentication with JWT tokens
- 🔐 Secure authentication with Better Auth (email/password, SSO, OIDC)
- 📊 Real-time dashboard with activity logs
- ⏱️ Scheduled automatic mirroring
- 🐳 Dockerized with multi-arch support (AMD64/ARM64)

74
UPGRADE.md Normal file
View File

@@ -0,0 +1,74 @@
# Upgrade Guide
## Upgrading to v3.0
> **⚠️ IMPORTANT**: v3.0 requires a fresh start. There is no automated migration from v2.x to v3.0.
### Why No Migration?
v3.0 introduces fundamental changes to the application architecture:
- **Authentication**: Switched from JWT to Better Auth
- **Database**: Now uses Drizzle ORM with proper migrations
- **Security**: All tokens are now encrypted
- **Features**: Added SSO support and OIDC provider functionality
Due to these extensive changes, we recommend starting fresh with v3.0 for the best experience.
### Upgrade Steps
1. **Stop your v2.x container**
```bash
docker stop gitea-mirror
docker rm gitea-mirror
```
2. **Backup your v2.x data (optional)**
```bash
# If you want to keep your v2 data for reference
docker run --rm -v gitea-mirror-data:/data -v $(pwd):/backup alpine tar czf /backup/gitea-mirror-v2-backup.tar.gz -C /data .
```
3. **Create a new volume for v3**
```bash
docker volume create gitea-mirror-v3-data
```
4. **Run v3 with the new volume**
```bash
docker run -d \
--name gitea-mirror \
-p 4321:4321 \
-v gitea-mirror-v3-data:/app/data \
-e BETTER_AUTH_SECRET=your-secret-key \
-e ENCRYPTION_SECRET=your-encryption-key \
arunavo4/gitea-mirror:latest
```
5. **Set up your configuration again**
- Navigate to http://localhost:4321
- Create a new admin account
- Re-enter your GitHub and Gitea credentials
- Configure your mirror settings
### What Happens to My Existing Mirrors?
Your existing mirrors in Gitea are **not affected**. The application will:
- Recognize existing repositories when you re-import
- Skip creating duplicates
- Resume normal mirror operations
### Environment Variable Changes
v3.0 uses different environment variables:
| v2.x | v3.0 | Notes |
|------|------|-------|
| `JWT_SECRET` | `BETTER_AUTH_SECRET` | Required for session management |
| - | `ENCRYPTION_SECRET` | New - required for token encryption |
### Need Help?
If you have questions about upgrading:
1. Check the [README](README.md) for v3 setup instructions
2. Review your v2 configuration before upgrading
3. Open an issue if you encounter problems

77
docker-entrypoint.sh
View File

@@ -269,82 +269,7 @@ else
bun scripts/manage-db.ts fix
fi
# Run database migrations
echo "Running database migrations..."
# Update mirror_jobs table with new columns for resilience
if [ -f "dist/scripts/update-mirror-jobs-table.js" ]; then
echo "Updating mirror_jobs table..."
bun dist/scripts/update-mirror-jobs-table.js
elif [ -f "scripts/update-mirror-jobs-table.ts" ]; then
echo "Updating mirror_jobs table using TypeScript script..."
bun scripts/update-mirror-jobs-table.ts
else
echo "Warning: Could not find mirror_jobs table update script."
fi
# Run v3 migrations if needed
echo "Checking for v3 migrations..."
# Check if we need to run Better Auth migration (check if accounts table exists)
if ! sqlite3 /app/data/gitea-mirror.db "SELECT name FROM sqlite_master WHERE type='table' AND name='accounts';" | grep -q accounts; then
echo "🔄 v3 Migration: Creating Better Auth tables..."
# Create Better Auth tables
sqlite3 /app/data/gitea-mirror.db <<EOF
CREATE TABLE IF NOT EXISTS accounts (
id TEXT PRIMARY KEY,
userId TEXT NOT NULL,
accountId TEXT NOT NULL,
providerId TEXT NOT NULL,
accessToken TEXT,
refreshToken TEXT,
expiresAt INTEGER,
password TEXT,
createdAt INTEGER NOT NULL,
updatedAt INTEGER NOT NULL,
FOREIGN KEY (userId) REFERENCES users(id)
);
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
userId TEXT NOT NULL,
token TEXT NOT NULL,
expiresAt INTEGER NOT NULL,
createdAt INTEGER NOT NULL,
updatedAt INTEGER NOT NULL,
FOREIGN KEY (userId) REFERENCES users(id)
);
CREATE TABLE IF NOT EXISTS verification_tokens (
id TEXT PRIMARY KEY,
identifier TEXT NOT NULL,
token TEXT NOT NULL,
expires INTEGER NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_accounts_userId ON accounts(userId);
CREATE INDEX IF NOT EXISTS idx_sessions_token ON sessions(token);
CREATE INDEX IF NOT EXISTS idx_verification_identifier_token ON verification_tokens(identifier, token);
EOF
fi
# Run Better Auth user migration
if [ -f "dist/scripts/migrate-better-auth.js" ]; then
echo "🔄 v3 Migration: Migrating users to Better Auth..."
bun dist/scripts/migrate-better-auth.js
elif [ -f "scripts/migrate-better-auth.ts" ]; then
echo "🔄 v3 Migration: Migrating users to Better Auth..."
bun scripts/migrate-better-auth.ts
fi
# Run token encryption migration
if [ -f "dist/scripts/migrate-tokens-encryption.js" ]; then
echo "🔄 v3 Migration: Encrypting stored tokens..."
bun dist/scripts/migrate-tokens-encryption.js
elif [ -f "scripts/migrate-tokens-encryption.ts" ]; then
echo "🔄 v3 Migration: Encrypting stored tokens..."
bun scripts/migrate-tokens-encryption.ts
fi
echo "Database exists, checking integrity..."
fi
# Extract version from package.json and set as environment variable

2
package.json
View File

@@ -22,8 +22,6 @@
"db:pull": "bun drizzle-kit pull",
"db:check": "bun drizzle-kit check",
"db:studio": "bun drizzle-kit studio",
"migrate:better-auth": "bun scripts/migrate-to-better-auth.ts",
"migrate:encrypt-tokens": "bun scripts/migrate-tokens-encryption.ts",
"startup-recovery": "bun scripts/startup-recovery.ts",
"startup-recovery-force": "bun scripts/startup-recovery.ts --force",
"test-recovery": "bun scripts/test-recovery.ts",

100
scripts/migrate-better-auth.ts
View File

@@ -1,100 +0,0 @@
#!/usr/bin/env bun
import { db } from "../src/lib/db";
import { accounts } from "../src/lib/db/schema";
import { sql } from "drizzle-orm";
console.log("🔄 Starting Better Auth migration...");
async function migrateToBetterAuth() {
try {
// Check if migration is needed
const existingAccounts = await db.select().from(accounts).limit(1);
if (existingAccounts.length > 0) {
console.log("✓ Better Auth migration already completed");
return;
}
// Check if we have old users table with passwords
// This query checks if password column exists in users table
const hasPasswordColumn = await db.get<{ count: number }>(
sql`SELECT COUNT(*) as count FROM pragma_table_info('users') WHERE name = 'password'`
);
if (!hasPasswordColumn || hasPasswordColumn.count === 0) {
console.log("ℹ️ Users table doesn't have password column - migration may have already been done");
// Check if we have any users without accounts
const usersWithoutAccounts = await db.all<{ id: string; email: string }>(
sql`SELECT u.id, u.email FROM users u LEFT JOIN accounts a ON u.id = a.user_id WHERE a.id IS NULL`
);
if (usersWithoutAccounts.length === 0) {
console.log("✓ All users have accounts - migration complete");
return;
}
console.log(`⚠️ Found ${usersWithoutAccounts.length} users without accounts - they may need to reset passwords`);
return;
}
// Get all users with password hashes using raw SQL since the schema doesn't have password
const allUsersWithPasswords = await db.all<{ id: string; email: string; username: string; password: string }>(
sql`SELECT id, email, username, password FROM users WHERE password IS NOT NULL`
);
if (allUsersWithPasswords.length === 0) {
console.log("ℹ️ No users with passwords to migrate");
return;
}
console.log(`📊 Found ${allUsersWithPasswords.length} users to migrate`);
// Migrate each user
for (const user of allUsersWithPasswords) {
try {
// Create Better Auth account entry
await db.insert(accounts).values({
id: crypto.randomUUID(),
userId: user.id,
accountId: user.email, // Use email as account ID
providerId: "credential", // Better Auth credential provider
providerUserId: null,
accessToken: null,
refreshToken: null,
expiresAt: null,
password: user.password, // Move password hash to accounts table
createdAt: new Date(),
updatedAt: new Date()
});
console.log(`✓ Migrated user: ${user.email}`);
} catch (error) {
console.error(`❌ Failed to migrate user ${user.email}:`, error);
// Continue with other users even if one fails
}
}
// Remove password column from users table if it exists
console.log("🔄 Cleaning up old password column...");
try {
// SQLite doesn't support DROP COLUMN directly, so we need to recreate the table
// For now, we'll just leave it as is since it's not harmful
console.log("ℹ️ Password column left in users table for compatibility");
} catch (error) {
console.error("⚠️ Could not remove password column:", error);
}
console.log("✅ Better Auth migration completed successfully");
// Verify migration
const migratedAccounts = await db.select().from(accounts);
console.log(`📊 Total accounts after migration: ${migratedAccounts.length}`);
} catch (error) {
console.error("❌ Better Auth migration failed:", error);
process.exit(1);
}
}
// Run migration
migrateToBetterAuth();

87
scripts/migrate-to-better-auth.ts
View File

@@ -1,87 +0,0 @@
#!/usr/bin/env bun
import { db, users, accounts } from "../src/lib/db";
import { eq } from "drizzle-orm";
import { v4 as uuidv4 } from "uuid";
/**
* Migrate existing users to Better Auth schema
*
* This script:
* 1. Moves existing password hashes from users table to accounts table
* 2. Updates user data to match Better Auth requirements
* 3. Creates credential accounts for existing users
*/
async function migrateUsers() {
console.log("🔄 Starting user migration to Better Auth...");
try {
// Get all existing users
const existingUsers = await db.select().from(users);
if (existingUsers.length === 0) {
console.log("✅ No users to migrate");
return;
}
console.log(`Found ${existingUsers.length} users to migrate`);
for (const user of existingUsers) {
console.log(`\nMigrating user: ${user.username} (${user.email})`);
// Check if user already has a credential account
const existingAccount = await db
.select()
.from(accounts)
.where(
eq(accounts.userId, user.id) &&
eq(accounts.providerId, "credential")
)
.limit(1);
if (existingAccount.length > 0) {
console.log("✓ User already migrated");
continue;
}
// Create credential account with existing password hash
const accountId = uuidv4();
await db.insert(accounts).values({
id: accountId,
accountId: accountId,
userId: user.id,
providerId: "credential",
providerUserId: user.email, // Use email as provider user ID
// password: user.password, // Password is not in users table anymore
createdAt: user.createdAt,
updatedAt: user.updatedAt,
});
console.log("✓ Created credential account");
// Update user name field if it's null (Better Auth uses 'name' field)
// Note: Better Auth expects a 'name' field, but we're using username
// This is handled by our additional fields configuration
}
console.log("\n✅ User migration completed successfully!");
// Summary
const migratedAccounts = await db
.select()
.from(accounts)
.where(eq(accounts.providerId, "credential"));
console.log(`\nMigration Summary:`);
console.log(`- Total users: ${existingUsers.length}`);
console.log(`- Migrated accounts: ${migratedAccounts.length}`);
} catch (error) {
console.error("❌ Migration failed:", error);
process.exit(1);
}
}
// Run migration
migrateUsers();

135
scripts/migrate-tokens-encryption.ts
View File

@@ -1,135 +0,0 @@
#!/usr/bin/env bun
/**
* Migration script to encrypt existing GitHub and Gitea tokens in the database
* Run with: bun run scripts/migrate-tokens-encryption.ts
*/
import { db, configs } from "../src/lib/db";
import { eq } from "drizzle-orm";
import { encrypt, isEncrypted, migrateToken } from "../src/lib/utils/encryption";
async function migrateTokens() {
console.log("Starting token encryption migration...");
try {
// Fetch all configs
const allConfigs = await db.select().from(configs);
console.log(`Found ${allConfigs.length} configurations to check`);
let migratedCount = 0;
let skippedCount = 0;
let errorCount = 0;
for (const config of allConfigs) {
try {
let githubUpdated = false;
let giteaUpdated = false;
// Parse configs
const githubConfig = typeof config.githubConfig === "string"
? JSON.parse(config.githubConfig)
: config.githubConfig;
const giteaConfig = typeof config.giteaConfig === "string"
? JSON.parse(config.giteaConfig)
: config.giteaConfig;
// Check and migrate GitHub token
if (githubConfig.token) {
if (!isEncrypted(githubConfig.token)) {
console.log(`Encrypting GitHub token for config ${config.id} (user: ${config.userId})`);
githubConfig.token = encrypt(githubConfig.token);
githubUpdated = true;
} else {
console.log(`GitHub token already encrypted for config ${config.id}`);
}
}
// Check and migrate Gitea token
if (giteaConfig.token) {
if (!isEncrypted(giteaConfig.token)) {
console.log(`Encrypting Gitea token for config ${config.id} (user: ${config.userId})`);
giteaConfig.token = encrypt(giteaConfig.token);
giteaUpdated = true;
} else {
console.log(`Gitea token already encrypted for config ${config.id}`);
}
}
// Update config if any tokens were migrated
if (githubUpdated || giteaUpdated) {
await db
.update(configs)
.set({
githubConfig,
giteaConfig,
updatedAt: new Date(),
})
.where(eq(configs.id, config.id));
migratedCount++;
console.log(`✓ Config ${config.id} updated successfully`);
} else {
skippedCount++;
}
} catch (error) {
errorCount++;
console.error(`✗ Error processing config ${config.id}:`, error);
}
}
console.log("\n=== Migration Summary ===");
console.log(`Total configs: ${allConfigs.length}`);
console.log(`Migrated: ${migratedCount}`);
console.log(`Skipped (already encrypted): ${skippedCount}`);
console.log(`Errors: ${errorCount}`);
if (errorCount > 0) {
console.error("\n⚠️ Some configs failed to migrate. Please check the errors above.");
process.exit(1);
} else {
console.log("\n✅ Token encryption migration completed successfully!");
}
} catch (error) {
console.error("Fatal error during migration:", error);
process.exit(1);
}
}
// Verify environment setup
function verifyEnvironment() {
const requiredEnvVars = ["ENCRYPTION_SECRET", "JWT_SECRET", "BETTER_AUTH_SECRET"];
const availableSecrets = requiredEnvVars.filter(varName => process.env[varName]);
if (availableSecrets.length === 0) {
console.error("❌ No encryption secret found!");
console.error("Please set one of the following environment variables:");
console.error(" - ENCRYPTION_SECRET (recommended)");
console.error(" - JWT_SECRET");
console.error(" - BETTER_AUTH_SECRET");
process.exit(1);
}
console.log(`Using encryption secret from: ${availableSecrets[0]}`);
}
// Main execution
async function main() {
console.log("=== Gitea Mirror Token Encryption Migration ===\n");
// Verify environment
verifyEnvironment();
// Run migration
await migrateTokens();
process.exit(0);
}
main().catch((error) => {
console.error("Unexpected error:", error);
process.exit(1);
});