Compare commits

...

17 Commits

Author SHA1 Message Date
Arunavo Ray
832b57538d chore: bump version to 2.11.0 2025-05-28 14:08:45 +05:30
Arunavo Ray
415bff8e41 feat: enhance Organizations page with live refresh and fix repository breakdown bug
- Add live refresh functionality to Organizations page using the same pattern as Repositories and Activity Log pages
- Fix repository breakdown bug where public/private/fork counts disappeared after toggling mirroring
- Change toggle text from 'Include in mirroring' to 'Enable mirroring' for better clarity
- Automatically refresh organization data after mirroring starts to maintain breakdown visibility
- Clean up unused imports and variables for better code quality
2025-05-28 14:08:07 +05:30
Arunavo Ray
13c3ddea04 Added a small gap to Verison Info 2025-05-28 13:55:50 +05:30
Arunavo Ray
b917b30830 docs: add Docker bind mount vs named volume permission guidance
- Add new section 'Docker Volume Types and Permissions'
- Explain difference between named volumes and bind mounts
- Provide solution for bind mount permission issues (UID 1001)
- Clarify why named volumes are recommended and used in official docker-compose.yml
- Address SQLite permission errors in Docker environments using bind mounts

Addresses issue reported by user using bind mounts in Portainer.
2025-05-28 13:37:07 +05:30
Arunavo Ray
b34ed5595b chore: bump version to 2.10.0 2025-05-28 13:27:04 +05:30
Arunavo Ray
cbc11155ef fix: resolve organizations getting stuck on mirroring status when empty
- Fixed mirrorGitHubOrgToGitea function to properly handle empty organizations
- Organizations with no repositories now transition from 'mirroring' to 'mirrored' status
- Enhanced logging with clearer messages for empty organization processing
- Improved activity log details to distinguish between empty and non-empty orgs
- Added comprehensive test coverage for empty organization scenarios
- Ensures consistent status lifecycle for all organizations regardless of repository count
2025-05-28 13:26:20 +05:30
Arunavo Ray
941f61830f feat: implement comprehensive auto-save for all config forms and remove manual save button
- Add auto-save functionality to all GitHub config form fields (text inputs and checkboxes)
- Add auto-save functionality to all Gitea config form fields (text inputs and select dropdown)
- Extend existing auto-save pattern to cover text inputs with 500ms debounce
- Remove Save Configuration button and related manual save logic
- Update Import GitHub Data button to depend on form validation instead of saved state
- Remove isConfigSaved dependency from all auto-save functions for immediate activation
- Add proper cleanup for all auto-save timeouts on component unmount
- Maintain silent auto-save operation without intrusive notifications

All configuration changes now auto-save seamlessly, providing a better UX while maintaining data consistency and error handling.
2025-05-28 13:17:48 +05:30
Arunavo Ray
5b60cffaae Add fork tags to repository UI and enhance organization cards with repository breakdown
- Add fork tags to repository table and dashboard list components
- Display 'Fork' badge for repositories where isForked is true
- Enhance organization cards to show breakdown of public, private, and fork repositories
- Update organization API to respect user configuration filters (private repos, forks)
- Add visual indicators with colored dots for each repository type
- Ensure consistent filtering between repository and organization APIs
- Fix issue where private repositories weren't showing due to configuration filtering
2025-05-28 12:53:32 +05:30
Arunavo Ray
ede5b4dbe8 feat: enhance toast error messages with structured parsing
- Add parseErrorMessage() utility to parse JSON error responses
- Add showErrorToast() helper for consistent error display
- Update all toast.error calls to use structured error parsing
- Support multiple error formats: error+troubleshooting, title+description, message+details
- Enhance apiRequest() to support both 'body' and 'data' properties
- Add comprehensive unit tests for error parsing functionality
- Improve user experience with clear, actionable error messages

Fixes structured error messages from Gitea API responses that were showing as raw JSON
2025-05-28 11:11:28 +05:30
Arunavo Ray
99336e2607 chore: bump version to 2.9.2 2025-05-28 10:15:43 +05:30
Arunavo Ray
cba421d606 feat: enhance error logging for better debugging of JSON parsing issues
- Add comprehensive error logging in mirror-repo API endpoint
- Enhance HTTP client error handling with detailed response information
- Improve concurrency utility error reporting with context
- Add specific detection and guidance for JSON parsing errors
- Include troubleshooting information in error responses
- Update tests to accommodate enhanced logging

This will help users diagnose issues like 'JSON Parse error: Unexpected EOF'
by providing detailed information about what responses are being received
from the Gitea API and what might be causing the failures.
2025-05-28 10:13:41 +05:30
Arunavo Ray
c4b9a82806 chore: bump version to 2.9.1 2025-05-28 10:00:14 +05:30
Arunavo Ray
38e0fb33b9 fix: resolve JSON parsing error and standardize HTTP client usage
- Fix JSON parsing error in getOrCreateGiteaOrg function (#19)
  - Add content-type validation before JSON parsing
  - Add response cloning for better error debugging
  - Enhance error messages with actual response content
  - Add comprehensive logging for troubleshooting

- Standardize HTTP client usage across codebase
  - Create new http-client.ts utility with consistent error handling
  - Replace all superagent calls with fetch-based functions
  - Replace all axios calls with fetch-based functions
  - Remove superagent, axios, and @types/superagent dependencies
  - Update tests to mock new HTTP client
  - Maintain backward compatibility

- Benefits:
  - Smaller bundle size (removed 3 HTTP client libraries)
  - Better performance (leveraging Bun's optimized fetch)
  - Consistent error handling across all HTTP operations
  - Improved debugging with detailed error messages
  - Easier maintenance with single HTTP client pattern
2025-05-28 09:56:59 +05:30
Arunavo Ray
22a4b71653 docs: add SQLite permission troubleshooting for direct installation
- Add new section 'Database Permissions for Direct Installation' to README
- Explain common SQLite permission errors when running without Docker
- Provide secure permission fixes (chmod 755/644 instead of 777)
- Clarify why Docker deployment avoids these issues
- Recommend Docker/Docker Compose as preferred deployment method

Addresses permission issues reported by users running the application directly on their systems.
2025-05-28 09:21:01 +05:30
ARUNAVO RAY
52568eda36 Merge pull request #21 from arunavo4/18-meta-docs-links-are-broken
fix: correct broken documentation links in README
2025-05-28 09:11:59 +05:30
Arunavo Ray
a84191f0a5 fix: correct broken documentation links in README
- Fix Quick Start Guide link to point to src/content/docs/quickstart.md
- Fix Configuration Guide link to point to src/content/docs/configuration.md
- Links were previously pointing to non-existent docs/ directory
2025-05-28 09:10:38 +05:30
Arunavo Ray
33829eda20 fix: update image sizes in README for better display on dashboard 2025-05-25 11:08:36 +05:30
35 changed files with 1433 additions and 914 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 140 KiB

After

Width:  |  Height:  |  Size: 816 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 164 KiB

After

Width:  |  Height:  |  Size: 945 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 194 KiB

After

Width:  |  Height:  |  Size: 943 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 784 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 170 KiB

After

Width:  |  Height:  |  Size: 970 KiB

View File

@@ -30,7 +30,7 @@ sudo LOCAL_REPO_DIR=~/Development/gitea-mirror ./scripts/gitea-mirror-lxc-local.
See the [LXC Container Deployment Guide](scripts/README-lxc.md).
<p align="center">
<img src=".github/assets/dashboard.png" alt="Dashboard" width="80%"/>
<img src=".github/assets/dashboard.png" alt="Dashboard" width="full"/>
</p>
## ✨ Features
@@ -50,12 +50,12 @@ See the [LXC Container Deployment Guide](scripts/README-lxc.md).
## 📸 Screenshots
<p align="center">
<img src=".github/assets/repositories.png" width="45%"/>
<img src=".github/assets/organisations.png" width="45%"/>
<img src=".github/assets/repositories.png" width="49%"/>
<img src=".github/assets/organisations.png" width="49%"/>
</p>
<p align="center">
<img src=".github/assets/configuration.png" width="45%"/>
<img src=".github/assets/activity.png" width="45%"/>
<img src=".github/assets/configuration.png" width="49%"/>
<img src=".github/assets/activity.png" width="49%"/>
</p>
### Dashboard
@@ -69,7 +69,7 @@ Easily configure your GitHub and Gitea connections, set up automatic mirroring s
## Getting Started
See the [Quick Start Guide](docs/quickstart.md) for detailed instructions on getting up and running quickly.
See the [Quick Start Guide](src/content/docs/quickstart.md) for detailed instructions on getting up and running quickly.
### Prerequisites
@@ -282,9 +282,39 @@ bun run reset-users
bun run check-db
```
##### Database Permissions for Direct Installation
> [!IMPORTANT]
> **If you're running the application directly** (not using Docker), you may encounter SQLite permission errors. This is because SQLite requires both read/write access to the database file and write access to the directory containing the database.
**Common Error:**
```
Error: [ERROR] SQLiteError: attempt to write a readonly database
```
**Solution:**
```bash
# Ensure the data directory exists and has proper permissions
mkdir -p data
chmod 755 data
# If the database file already exists, ensure it's writable
chmod 644 data/gitea-mirror.db
# Make sure the user running the application owns the data directory
chown -R $(whoami) data/
```
**Why Docker doesn't have this issue:**
- Docker containers run with a dedicated user (`gitea-mirror`) that owns the `/app/data` directory
- The container setup ensures proper permissions are set during image build
- Volume mounts are handled by Docker with appropriate permissions
**Recommended approach:** Use Docker or Docker Compose for deployment to avoid permission issues entirely.
### Configuration
Gitea Mirror can be configured through environment variables or through the web UI. See the [Configuration Guide](docs/configuration.md) for more details.
Gitea Mirror can be configured through environment variables or through the web UI. See the [Configuration Guide](src/content/docs/configuration.md) for more details.
Key configuration options include:
@@ -483,6 +513,36 @@ Try the following steps:
>
> This setup provides a complete containerized deployment for the Gitea Mirror application.
#### Docker Volume Types and Permissions
> [!IMPORTANT]
> **Named Volumes vs Bind Mounts**: If you encounter SQLite permission errors even when using Docker, check your volume configuration:
**✅ Named Volumes (Recommended):**
```yaml
volumes:
- gitea-mirror-data:/app/data # Docker manages permissions automatically
```
**⚠️ Bind Mounts (Requires Manual Permission Setup):**
```yaml
volumes:
- /host/path/to/data:/app/data # Host filesystem permissions apply
```
**If using bind mounts**, ensure the host directory is owned by UID 1001 (the `gitea-mirror` user):
```bash
# Set correct ownership for bind mount
sudo chown -R 1001:1001 /host/path/to/data
sudo chmod -R 755 /host/path/to/data
```
**Why named volumes work better:**
- Docker automatically handles permissions
- Better portability across different hosts
- No manual permission setup required
- Used by our official docker-compose.yml
#### Database Maintenance

607
bun.lock

File diff suppressed because it is too large Load Diff

View File

@@ -1,19 +1,18 @@
{
"name": "gitea-mirror",
"type": "module",
"version": "2.9.0",
"version": "2.11.0",
"engines": {
"bun": ">=1.2.9"
},
"scripts": {
"setup": "bun install && bun run manage-db init && bun run update-db",
"setup": "bun install && bun run manage-db init",
"dev": "bunx --bun astro dev",
"dev:clean": "bun run cleanup-db && bun run manage-db init && bun run update-db && bunx --bun astro dev",
"dev:clean": "bun run cleanup-db && bun run manage-db init && bunx --bun astro dev",
"build": "bunx --bun astro build",
"cleanup-db": "rm -f gitea-mirror.db data/gitea-mirror.db",
"manage-db": "bun scripts/manage-db.ts",
"init-db": "bun scripts/manage-db.ts init",
"update-db": "bun scripts/update-mirror-jobs-table.ts",
"check-db": "bun scripts/manage-db.ts check",
"fix-db": "bun scripts/manage-db.ts fix",
"reset-users": "bun scripts/manage-db.ts reset-users",
@@ -26,7 +25,7 @@
"test-shutdown-cleanup": "bun scripts/test-graceful-shutdown.ts --cleanup",
"preview": "bunx --bun astro preview",
"start": "bun dist/server/entry.mjs",
"start:fresh": "bun run cleanup-db && bun run manage-db init && bun run update-db && bun dist/server/entry.mjs",
"start:fresh": "bun run cleanup-db && bun run manage-db init && bun dist/server/entry.mjs",
"test": "bun test",
"test:watch": "bun test --watch",
"test:coverage": "bun test --coverage",
@@ -55,7 +54,6 @@
"@types/react": "^19.1.4",
"@types/react-dom": "^19.1.5",
"astro": "^5.7.13",
"axios": "^1.9.0",
"bcryptjs": "^3.0.2",
"canvas-confetti": "^1.9.3",
"class-variance-authority": "^0.7.1",
@@ -70,7 +68,6 @@
"react-dom": "^19.1.0",
"react-icons": "^5.5.0",
"sonner": "^2.0.3",
"superagent": "^10.2.1",
"tailwind-merge": "^3.3.0",
"tailwindcss": "^4.1.7",
"tw-animate-css": "^1.3.0",
@@ -82,7 +79,6 @@
"@testing-library/react": "^16.3.0",
"@types/bcryptjs": "^3.0.0",
"@types/jsonwebtoken": "^9.0.9",
"@types/superagent": "^8.1.9",
"@types/uuid": "^10.0.0",
"@vitejs/plugin-react": "^4.4.1",
"jsdom": "^26.1.0",

View File

@@ -16,7 +16,7 @@ import {
DialogTitle,
DialogTrigger,
} from '../ui/dialog';
import { apiRequest, formatDate } from '@/lib/utils';
import { apiRequest, formatDate, showErrorToast } from '@/lib/utils';
import { useAuth } from '@/hooks/useAuth';
import type { MirrorJob } from '@/lib/db/schema';
import type { ActivityApiResponse } from '@/types/activities';
@@ -155,7 +155,7 @@ export function ActivityLog() {
if (!res.success) {
// Only show error toast for manual refreshes to avoid spam during live updates
if (!isLiveRefresh) {
toast.error(res.message ?? 'Failed to fetch activities.');
showErrorToast(res.message ?? 'Failed to fetch activities.', toast);
}
return false;
}
@@ -184,9 +184,7 @@ export function ActivityLog() {
if (isMountedRef.current) {
// Only show error toast for manual refreshes to avoid spam during live updates
if (!isLiveRefresh) {
toast.error(
err instanceof Error ? err.message : 'Failed to fetch activities.',
);
showErrorToast(err, toast);
}
}
return false;
@@ -313,7 +311,6 @@ export function ActivityLog() {
setIsInitialLoading(true);
setShowCleanupDialog(false);
// Use fetch directly to avoid potential axios issues
const response = await fetch('/api/activities/cleanup', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
@@ -332,11 +329,11 @@ export function ActivityLog() {
setActivities([]);
toast.success(`All activities cleaned up successfully. Deleted ${res.result.mirrorJobsDeleted} mirror jobs and ${res.result.eventsDeleted} events.`);
} else {
toast.error(res.error || 'Failed to cleanup activities.');
showErrorToast(res.error || 'Failed to cleanup activities.', toast);
}
} catch (error) {
console.error('Error cleaning up activities:', error);
toast.error(error instanceof Error ? error.message : 'Failed to cleanup activities.');
showErrorToast(error, toast);
} finally {
setIsInitialLoading(false);
}

View File

@@ -6,6 +6,7 @@ import { Button } from '@/components/ui/button';
import { Card, CardContent, CardDescription, CardFooter, CardHeader, CardTitle } from '@/components/ui/card';
import { SiGitea } from 'react-icons/si';
import { toast, Toaster } from 'sonner';
import { showErrorToast } from '@/lib/utils';
import { FlipHorizontal } from 'lucide-react';
export function LoginForm() {
@@ -45,10 +46,10 @@ export function LoginForm() {
window.location.href = '/';
}, 1000);
} else {
toast.error(data.error || 'Login failed. Please try again.');
showErrorToast(data.error || 'Login failed. Please try again.', toast);
}
} catch (error) {
toast.error('An error occurred while logging in. Please try again.');
showErrorToast(error, toast);
} finally {
setIsLoading(false);
}

View File

@@ -5,6 +5,7 @@ import { Button } from '@/components/ui/button';
import { Card, CardContent, CardDescription, CardFooter, CardHeader, CardTitle } from '@/components/ui/card';
import { GitMerge } from 'lucide-react';
import { toast, Toaster } from 'sonner';
import { showErrorToast } from '@/lib/utils';
export function SignupForm() {
const [isLoading, setIsLoading] = useState(false);
@@ -51,10 +52,10 @@ export function SignupForm() {
window.location.href = '/';
}, 1500);
} else {
toast.error(data.error || 'Failed to create account. Please try again.');
showErrorToast(data.error || 'Failed to create account. Please try again.', toast);
}
} catch (error) {
toast.error('An error occurred while creating your account. Please try again.');
showErrorToast(error, toast);
} finally {
setIsLoading(false);
}

View File

@@ -14,7 +14,7 @@ import type {
} from '@/types/config';
import { Button } from '../ui/button';
import { useAuth } from '@/hooks/useAuth';
import { apiRequest } from '@/lib/utils';
import { apiRequest, showErrorToast } from '@/lib/utils';
import { RefreshCw } from 'lucide-react';
import { toast } from 'sonner';
import { Skeleton } from '@/components/ui/skeleton';
@@ -56,14 +56,18 @@ export function ConfigTabs() {
retentionDays: 604800, // 7 days in seconds
},
});
const { user, refreshUser } = useAuth();
const { user } = useAuth();
const [isLoading, setIsLoading] = useState(true);
const [isSyncing, setIsSyncing] = useState<boolean>(false);
const [isConfigSaved, setIsConfigSaved] = useState<boolean>(false);
const [isAutoSavingSchedule, setIsAutoSavingSchedule] = useState<boolean>(false);
const [isAutoSavingCleanup, setIsAutoSavingCleanup] = useState<boolean>(false);
const [isAutoSavingGitHub, setIsAutoSavingGitHub] = useState<boolean>(false);
const [isAutoSavingGitea, setIsAutoSavingGitea] = useState<boolean>(false);
const autoSaveScheduleTimeoutRef = useRef<NodeJS.Timeout | null>(null);
const autoSaveCleanupTimeoutRef = useRef<NodeJS.Timeout | null>(null);
const autoSaveGitHubTimeoutRef = useRef<NodeJS.Timeout | null>(null);
const autoSaveGiteaTimeoutRef = useRef<NodeJS.Timeout | null>(null);
const isConfigFormValid = (): boolean => {
const { githubConfig, giteaConfig } = config;
@@ -109,47 +113,9 @@ export function ConfigTabs() {
}
};
const handleSaveConfig = async () => {
if (!user?.id) return;
const reqPayload: SaveConfigApiRequest = {
userId: user.id,
githubConfig: config.githubConfig,
giteaConfig: config.giteaConfig,
scheduleConfig: config.scheduleConfig,
cleanupConfig: config.cleanupConfig,
};
try {
const response = await fetch('/api/config', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(reqPayload),
});
const result: SaveConfigApiResponse = await response.json();
if (result.success) {
await refreshUser();
setIsConfigSaved(true);
// Invalidate config cache so other components get fresh data
invalidateConfigCache();
toast.success(
'Configuration saved successfully! Now import your GitHub data to begin.',
);
} else {
toast.error(
`Failed to save configuration: ${result.message || 'Unknown error'}`,
);
}
} catch (error) {
toast.error(
`An error occurred while saving the configuration: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
};
// Auto-save function specifically for schedule config changes
const autoSaveScheduleConfig = useCallback(async (scheduleConfig: ScheduleConfig) => {
if (!user?.id || !isConfigSaved) return; // Only auto-save if config was previously saved
if (!user?.id) return;
// Clear any existing timeout
if (autoSaveScheduleTimeoutRef.current) {
@@ -198,27 +164,22 @@ export function ConfigTabs() {
console.warn('Failed to fetch updated config after auto-save:', fetchError);
}
} else {
toast.error(
showErrorToast(
`Auto-save failed: ${result.message || 'Unknown error'}`,
{ duration: 3000 }
toast
);
}
} catch (error) {
toast.error(
`Auto-save error: ${
error instanceof Error ? error.message : String(error)
}`,
{ duration: 3000 }
);
showErrorToast(error, toast);
} finally {
setIsAutoSavingSchedule(false);
}
}, 500); // 500ms debounce
}, [user?.id, isConfigSaved, config.githubConfig, config.giteaConfig, config.cleanupConfig]);
}, [user?.id, config.githubConfig, config.giteaConfig, config.cleanupConfig]);
// Auto-save function specifically for cleanup config changes
const autoSaveCleanupConfig = useCallback(async (cleanupConfig: DatabaseCleanupConfig) => {
if (!user?.id || !isConfigSaved) return; // Only auto-save if config was previously saved
if (!user?.id) return;
// Clear any existing timeout
if (autoSaveCleanupTimeoutRef.current) {
@@ -266,23 +227,112 @@ export function ConfigTabs() {
console.warn('Failed to fetch updated config after auto-save:', fetchError);
}
} else {
toast.error(
showErrorToast(
`Auto-save failed: ${result.message || 'Unknown error'}`,
{ duration: 3000 }
toast
);
}
} catch (error) {
toast.error(
`Auto-save error: ${
error instanceof Error ? error.message : String(error)
}`,
{ duration: 3000 }
);
showErrorToast(error, toast);
} finally {
setIsAutoSavingCleanup(false);
}
}, 500); // 500ms debounce
}, [user?.id, isConfigSaved, config.githubConfig, config.giteaConfig, config.scheduleConfig]);
}, [user?.id, config.githubConfig, config.giteaConfig, config.scheduleConfig]);
// Auto-save function specifically for GitHub config changes
const autoSaveGitHubConfig = useCallback(async (githubConfig: GitHubConfig) => {
if (!user?.id) return;
// Clear any existing timeout
if (autoSaveGitHubTimeoutRef.current) {
clearTimeout(autoSaveGitHubTimeoutRef.current);
}
// Debounce the auto-save to prevent excessive API calls
autoSaveGitHubTimeoutRef.current = setTimeout(async () => {
setIsAutoSavingGitHub(true);
const reqPayload: SaveConfigApiRequest = {
userId: user.id!,
githubConfig: githubConfig,
giteaConfig: config.giteaConfig,
scheduleConfig: config.scheduleConfig,
cleanupConfig: config.cleanupConfig,
};
try {
const response = await fetch('/api/config', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(reqPayload),
});
const result: SaveConfigApiResponse = await response.json();
if (result.success) {
// Silent success - no toast for auto-save
// Invalidate config cache so other components get fresh data
invalidateConfigCache();
} else {
showErrorToast(
`Auto-save failed: ${result.message || 'Unknown error'}`,
toast
);
}
} catch (error) {
showErrorToast(error, toast);
} finally {
setIsAutoSavingGitHub(false);
}
}, 500); // 500ms debounce
}, [user?.id, config.giteaConfig, config.scheduleConfig, config.cleanupConfig]);
// Auto-save function specifically for Gitea config changes
const autoSaveGiteaConfig = useCallback(async (giteaConfig: GiteaConfig) => {
if (!user?.id) return;
// Clear any existing timeout
if (autoSaveGiteaTimeoutRef.current) {
clearTimeout(autoSaveGiteaTimeoutRef.current);
}
// Debounce the auto-save to prevent excessive API calls
autoSaveGiteaTimeoutRef.current = setTimeout(async () => {
setIsAutoSavingGitea(true);
const reqPayload: SaveConfigApiRequest = {
userId: user.id!,
githubConfig: config.githubConfig,
giteaConfig: giteaConfig,
scheduleConfig: config.scheduleConfig,
cleanupConfig: config.cleanupConfig,
};
try {
const response = await fetch('/api/config', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(reqPayload),
});
const result: SaveConfigApiResponse = await response.json();
if (result.success) {
// Silent success - no toast for auto-save
// Invalidate config cache so other components get fresh data
invalidateConfigCache();
} else {
showErrorToast(
`Auto-save failed: ${result.message || 'Unknown error'}`,
toast
);
}
} catch (error) {
showErrorToast(error, toast);
} finally {
setIsAutoSavingGitea(false);
}
}, 500); // 500ms debounce
}, [user?.id, config.githubConfig, config.scheduleConfig, config.cleanupConfig]);
// Cleanup timeouts on unmount
useEffect(() => {
@@ -293,6 +343,12 @@ export function ConfigTabs() {
if (autoSaveCleanupTimeoutRef.current) {
clearTimeout(autoSaveCleanupTimeoutRef.current);
}
if (autoSaveGitHubTimeoutRef.current) {
clearTimeout(autoSaveGitHubTimeoutRef.current);
}
if (autoSaveGiteaTimeoutRef.current) {
clearTimeout(autoSaveGiteaTimeoutRef.current);
}
};
}, []);
@@ -317,7 +373,7 @@ export function ConfigTabs() {
cleanupConfig:
response.cleanupConfig || config.cleanupConfig,
});
if (response.id) setIsConfigSaved(true);
}
} catch (error) {
console.warn(
@@ -414,10 +470,10 @@ export function ConfigTabs() {
<div className="flex gap-x-4">
<Button
onClick={handleImportGitHubData}
disabled={isSyncing || !isConfigSaved}
disabled={isSyncing || !isConfigFormValid()}
title={
!isConfigSaved
? 'Save configuration first'
!isConfigFormValid()
? 'Please fill all required GitHub and Gitea fields'
: isSyncing
? 'Import in progress'
: 'Import GitHub Data'
@@ -435,17 +491,6 @@ export function ConfigTabs() {
</>
)}
</Button>
<Button
onClick={handleSaveConfig}
disabled={!isConfigFormValid()}
title={
!isConfigFormValid()
? 'Please fill all required fields'
: 'Save Configuration'
}
>
Save Configuration
</Button>
</div>
</div>
@@ -463,6 +508,8 @@ export function ConfigTabs() {
: update,
}))
}
onAutoSave={autoSaveGitHubConfig}
isAutoSaving={isAutoSavingGitHub}
/>
<GiteaConfigForm
config={config.giteaConfig}
@@ -475,6 +522,8 @@ export function ConfigTabs() {
: update,
}))
}
onAutoSave={autoSaveGiteaConfig}
isAutoSaving={isAutoSavingGitea}
/>
</div>
<div className="flex gap-x-4">

View File

@@ -20,9 +20,11 @@ import { Tooltip, TooltipContent, TooltipTrigger } from "../ui/tooltip";
interface GitHubConfigFormProps {
config: GitHubConfig;
setConfig: React.Dispatch<React.SetStateAction<GitHubConfig>>;
onAutoSave?: (githubConfig: GitHubConfig) => Promise<void>;
isAutoSaving?: boolean;
}
export function GitHubConfigForm({ config, setConfig }: GitHubConfigFormProps) {
export function GitHubConfigForm({ config, setConfig, onAutoSave, isAutoSaving }: GitHubConfigFormProps) {
const [isLoading, setIsLoading] = useState(false);
const handleChange = (e: React.ChangeEvent<HTMLInputElement>) => {
@@ -43,10 +45,17 @@ export function GitHubConfigForm({ config, setConfig }: GitHubConfigFormProps) {
);
}
setConfig({
const newConfig = {
...config,
[name]: type === "checkbox" ? checked : value,
});
};
setConfig(newConfig);
// Auto-save for all field changes
if (onAutoSave) {
onAutoSave(newConfig);
}
};
const testConnection = async () => {

View File

@@ -21,19 +21,27 @@ import { toast } from "sonner";
interface GiteaConfigFormProps {
config: GiteaConfig;
setConfig: React.Dispatch<React.SetStateAction<GiteaConfig>>;
onAutoSave?: (giteaConfig: GiteaConfig) => Promise<void>;
isAutoSaving?: boolean;
}
export function GiteaConfigForm({ config, setConfig }: GiteaConfigFormProps) {
export function GiteaConfigForm({ config, setConfig, onAutoSave, isAutoSaving }: GiteaConfigFormProps) {
const [isLoading, setIsLoading] = useState(false);
const handleChange = (
e: React.ChangeEvent<HTMLInputElement | HTMLSelectElement>
) => {
const { name, value } = e.target;
setConfig({
const newConfig = {
...config,
[name]: value,
});
};
setConfig(newConfig);
// Auto-save for all field changes
if (onAutoSave) {
onAutoSave(newConfig);
}
};
const testConnection = async () => {

View File

@@ -5,7 +5,7 @@ import { GitFork, Clock, FlipHorizontal, Building2 } from "lucide-react";
import { useCallback, useEffect, useRef, useState } from "react";
import type { MirrorJob, Organization, Repository } from "@/lib/db/schema";
import { useAuth } from "@/hooks/useAuth";
import { apiRequest } from "@/lib/utils";
import { apiRequest, showErrorToast } from "@/lib/utils";
import type { DashboardApiResponse } from "@/types/dashboard";
import { useSSE } from "@/hooks/useSEE";
import { toast } from "sonner";
@@ -103,15 +103,11 @@ export function Dashboard() {
}
return true;
} else {
toast.error(response.error || "Error fetching dashboard data");
showErrorToast(response.error || "Error fetching dashboard data", toast);
return false;
}
} catch (error) {
toast.error(
error instanceof Error
? error.message
: "Error fetching dashboard data"
);
showErrorToast(error, toast);
return false;
} finally {
setIsLoading(false);

View File

@@ -81,6 +81,11 @@ export function RepositoryList({ repositories }: RepositoryListProps) {
Private
</span>
)}
{repo.isForked && (
<span className="rounded-full bg-muted px-2 py-0.5 text-xs">
Fork
</span>
)}
</div>
<div className="flex items-center gap-2 mt-1">
<span className="text-xs text-muted-foreground">

View File

@@ -37,7 +37,7 @@ export function VersionInfo() {
return (
<div className="text-xs text-muted-foreground text-center pt-2 pb-3 border-t border-border mt-2">
{versionInfo.updateAvailable ? (
<div className="flex flex-col">
<div className="flex flex-col gap-1">
<span>v{versionInfo.current}</span>
<span className="text-primary">v{versionInfo.latest} available</span>
</div>

View File

@@ -1,11 +1,11 @@
import { useCallback, useEffect, useState } from "react";
import { Button } from "@/components/ui/button";
import { Search, RefreshCw, FlipHorizontal, Plus } from "lucide-react";
import { Search, RefreshCw, FlipHorizontal } from "lucide-react";
import type { MirrorJob, Organization } from "@/lib/db/schema";
import { OrganizationList } from "./OrganizationsList";
import AddOrganizationDialog from "./AddOrganizationDialog";
import { useAuth } from "@/hooks/useAuth";
import { apiRequest } from "@/lib/utils";
import { apiRequest, showErrorToast } from "@/lib/utils";
import {
membershipRoleEnum,
type AddOrganizationApiRequest,
@@ -26,6 +26,7 @@ import { useFilterParams } from "@/hooks/useFilterParams";
import { toast } from "sonner";
import { useConfigStatus } from "@/hooks/useConfigStatus";
import { useNavigation } from "@/components/layout/MainLayout";
import { useLiveRefresh } from "@/hooks/useLiveRefresh";
export function Organization() {
const [organizations, setOrganizations] = useState<Organization[]>([]);
@@ -34,6 +35,7 @@ export function Organization() {
const { user } = useAuth();
const { isGitHubConfigured } = useConfigStatus();
const { navigationKey } = useNavigation();
const { registerRefreshCallback } = useLiveRefresh();
const { filter, setFilter } = useFilterParams({
searchTerm: "",
membershipRole: "",
@@ -62,19 +64,23 @@ export function Organization() {
onMessage: handleNewMessage,
});
const fetchOrganizations = useCallback(async () => {
const fetchOrganizations = useCallback(async (isLiveRefresh = false) => {
if (!user?.id) {
return false;
}
// Don't fetch organizations if GitHub is not configured
if (!isGitHubConfigured) {
setIsLoading(false);
if (!isLiveRefresh) {
setIsLoading(false);
}
return false;
}
try {
setIsLoading(true);
if (!isLiveRefresh) {
setIsLoading(true);
}
const response = await apiRequest<OrganizationsApiResponse>(
`/github/organizations?userId=${user.id}`,
@@ -87,27 +93,47 @@ export function Organization() {
setOrganizations(response.organizations);
return true;
} else {
toast.error(response.error || "Error fetching organizations");
if (!isLiveRefresh) {
toast.error(response.error || "Error fetching organizations");
}
return false;
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error fetching organizations"
);
if (!isLiveRefresh) {
toast.error(
error instanceof Error ? error.message : "Error fetching organizations"
);
}
return false;
} finally {
setIsLoading(false);
if (!isLiveRefresh) {
setIsLoading(false);
}
}
}, [user?.id, isGitHubConfigured]); // Only depend on user.id, not entire user object
useEffect(() => {
// Reset loading state when component becomes active
setIsLoading(true);
fetchOrganizations();
fetchOrganizations(false); // Manual refresh, not live
}, [fetchOrganizations, navigationKey]); // Include navigationKey to trigger on navigation
// Register with global live refresh system
useEffect(() => {
// Only register for live refresh if GitHub is configured
if (!isGitHubConfigured) {
return;
}
const unregister = registerRefreshCallback(() => {
fetchOrganizations(true); // Live refresh
});
return unregister;
}, [registerRefreshCallback, fetchOrganizations, isGitHubConfigured]);
const handleRefresh = async () => {
const success = await fetchOrganizations();
const success = await fetchOrganizations(false);
if (success) {
toast.success("Organizations refreshed successfully.");
}
@@ -140,6 +166,12 @@ export function Organization() {
return updated ? updated : org;
})
);
// Refresh organization data to get updated repository breakdown
// Use a small delay to allow the backend to process the mirroring request
setTimeout(() => {
fetchOrganizations(true);
}, 1000);
} else {
toast.error(response.error || "Error starting mirror job");
}
@@ -193,12 +225,10 @@ export function Organization() {
searchTerm: org,
}));
} else {
toast.error(response.error || "Error adding organization");
showErrorToast(response.error || "Error adding organization", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error adding organization"
);
showErrorToast(error, toast);
} finally {
setIsLoading(false);
}
@@ -250,24 +280,17 @@ export function Organization() {
})
);
} else {
toast.error(response.error || "Error starting mirror jobs");
showErrorToast(response.error || "Error starting mirror jobs", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error starting mirror jobs"
);
showErrorToast(error, toast);
} finally {
// Reset loading states - we'll let the SSE updates handle status changes
setLoadingOrgIds(new Set());
}
};
// Get unique organization names for combobox (since Organization has no owner field)
const ownerOptions = Array.from(
new Set(
organizations.map((org) => org.name).filter((v): v is string => !!v)
)
).sort();
return (
<div className="flex flex-col gap-y-8">

View File

@@ -118,10 +118,38 @@ export function OrganizationList({
</span>
</div>
<p className="text-sm text-muted-foreground mb-4">
{org.repositoryCount}{" "}
{org.repositoryCount === 1 ? "repository" : "repositories"}
</p>
<div className="text-sm text-muted-foreground mb-4">
<div className="flex items-center justify-between">
<span className="font-medium">
{org.repositoryCount}{" "}
{org.repositoryCount === 1 ? "repository" : "repositories"}
</span>
</div>
{(org.publicRepositoryCount !== undefined ||
org.privateRepositoryCount !== undefined ||
org.forkRepositoryCount !== undefined) && (
<div className="flex gap-4 mt-2 text-xs">
{org.publicRepositoryCount !== undefined && (
<span className="flex items-center gap-1">
<div className="h-2 w-2 rounded-full bg-green-500" />
{org.publicRepositoryCount} public
</span>
)}
{org.privateRepositoryCount !== undefined && org.privateRepositoryCount > 0 && (
<span className="flex items-center gap-1">
<div className="h-2 w-2 rounded-full bg-orange-500" />
{org.privateRepositoryCount} private
</span>
)}
{org.forkRepositoryCount !== undefined && org.forkRepositoryCount > 0 && (
<span className="flex items-center gap-1">
<div className="h-2 w-2 rounded-full bg-blue-500" />
{org.forkRepositoryCount} fork{org.forkRepositoryCount !== 1 ? 's' : ''}
</span>
)}
</div>
)}
</div>
<div className="flex items-center justify-between">
<div className="flex items-center">
@@ -144,7 +172,7 @@ export function OrganizationList({
htmlFor={`include-${org.id}`}
className="ml-2 text-sm select-none peer-disabled:cursor-not-allowed peer-disabled:opacity-70"
>
Include in mirroring
Enable mirroring
</label>
{isLoading && (

View File

@@ -9,7 +9,7 @@ import {
type RepositoryApiResponse,
type RepoStatus,
} from "@/types/Repository";
import { apiRequest } from "@/lib/utils";
import { apiRequest, showErrorToast } from "@/lib/utils";
import {
Select,
SelectContent,
@@ -108,16 +108,14 @@ export default function Repository() {
} else {
// Only show error toast for manual refreshes to avoid spam during live updates
if (!isLiveRefresh) {
toast.error(response.error || "Error fetching repositories");
showErrorToast(response.error || "Error fetching repositories", toast);
}
return false;
}
} catch (error) {
// Only show error toast for manual refreshes to avoid spam during live updates
if (!isLiveRefresh) {
toast.error(
error instanceof Error ? error.message : "Error fetching repositories"
);
showErrorToast(error, toast);
}
return false;
} finally {
@@ -184,12 +182,10 @@ export default function Repository() {
})
);
} else {
toast.error(response.error || "Error starting mirror job");
showErrorToast(response.error || "Error starting mirror job", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error starting mirror job"
);
showErrorToast(error, toast);
} finally {
setLoadingRepoIds((prev) => {
const newSet = new Set(prev);
@@ -248,12 +244,10 @@ export default function Repository() {
})
);
} else {
toast.error(response.error || "Error starting mirror jobs");
showErrorToast(response.error || "Error starting mirror jobs", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error starting mirror jobs"
);
showErrorToast(error, toast);
} finally {
// Reset loading states - we'll let the SSE updates handle status changes
setLoadingRepoIds(new Set());
@@ -287,12 +281,10 @@ export default function Repository() {
})
);
} else {
toast.error(response.error || "Error starting sync job");
showErrorToast(response.error || "Error starting sync job", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error starting sync job"
);
showErrorToast(error, toast);
} finally {
setLoadingRepoIds((prev) => {
const newSet = new Set(prev);
@@ -329,12 +321,10 @@ export default function Repository() {
})
);
} else {
toast.error(response.error || "Error retrying job");
showErrorToast(response.error || "Error retrying job", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error retrying job"
);
showErrorToast(error, toast);
} finally {
setLoadingRepoIds((prev) => {
const newSet = new Set(prev);
@@ -381,12 +371,10 @@ export default function Repository() {
searchTerm: repo,
}));
} else {
toast.error(response.error || "Error adding repository");
showErrorToast(response.error || "Error adding repository", toast);
}
} catch (error) {
toast.error(
error instanceof Error ? error.message : "Error adding repository"
);
showErrorToast(error, toast);
}
};

View File

@@ -249,6 +249,11 @@ export default function RepositoryTable({
Private
</span>
)}
{repo.isForked && (
<span className="ml-2 rounded-full bg-muted px-2 py-0.5 text-xs">
Fork
</span>
)}
</div>
{/* Owner */}

View File

@@ -152,6 +152,9 @@ export const organizationSchema = z.object({
errorMessage: z.string().optional(),
repositoryCount: z.number().default(0),
publicRepositoryCount: z.number().optional(),
privateRepositoryCount: z.number().optional(),
forkRepositoryCount: z.number().optional(),
createdAt: z.date().default(() => new Date()),
updatedAt: z.date().default(() => new Date()),

View File

@@ -1,6 +1,7 @@
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
import { Octokit } from "@octokit/rest";
import { repoStatusEnum } from "@/types/Repository";
import { getOrCreateGiteaOrg } from "./gitea";
// Mock the isRepoPresentInGitea function
const mockIsRepoPresentInGitea = mock(() => Promise.resolve(false));
@@ -27,23 +28,17 @@ mock.module("@/lib/helpers", () => {
};
});
// Mock superagent
mock.module("superagent", () => {
const mockPost = mock(() => ({
set: () => ({
set: () => ({
send: () => Promise.resolve({ body: { id: 123 } })
})
})
}));
const mockGet = mock(() => ({
set: () => Promise.resolve({ body: [] })
}));
// Mock http-client
mock.module("@/lib/http-client", () => {
return {
post: mockPost,
get: mockGet
httpPost: mock(() => Promise.resolve({ data: { id: 123 }, status: 200, statusText: 'OK', headers: new Headers() })),
httpGet: mock(() => Promise.resolve({ data: [], status: 200, statusText: 'OK', headers: new Headers() })),
HttpError: class MockHttpError extends Error {
constructor(message: string, public status: number, public statusText: string, public response?: string) {
super(message);
this.name = 'HttpError';
}
}
};
});
@@ -117,4 +112,182 @@ describe("Gitea Repository Mirroring", () => {
// Check that the function was called
expect(mirrorGithubRepoToGitea).toHaveBeenCalled();
});
test("getOrCreateGiteaOrg handles JSON parsing errors gracefully", async () => {
// Mock fetch to return invalid JSON
const originalFetch = global.fetch;
global.fetch = mock(async (url: string) => {
if (url.includes("/api/v1/orgs/")) {
// Mock response that looks successful but has invalid JSON
return {
ok: true,
status: 200,
headers: {
get: (name: string) => name === "content-type" ? "application/json" : null
},
json: () => Promise.reject(new Error("Unexpected token in JSON")),
text: () => Promise.resolve("Invalid JSON response"),
clone: function() {
return {
text: () => Promise.resolve("Invalid JSON response")
};
}
} as any;
}
return originalFetch(url);
});
const config = {
userId: "user-id",
giteaConfig: {
url: "https://gitea.example.com",
token: "gitea-token"
}
};
try {
await getOrCreateGiteaOrg({
orgName: "test-org",
config
});
// Should not reach here
expect(true).toBe(false);
} catch (error) {
// Should catch the JSON parsing error with a descriptive message
expect(error).toBeInstanceOf(Error);
expect((error as Error).message).toContain("Failed to parse JSON response from Gitea API");
} finally {
// Restore original fetch
global.fetch = originalFetch;
}
});
test("getOrCreateGiteaOrg handles non-JSON content-type gracefully", async () => {
// Mock fetch to return HTML instead of JSON
const originalFetch = global.fetch;
global.fetch = mock(async (url: string) => {
if (url.includes("/api/v1/orgs/")) {
return {
ok: true,
status: 200,
headers: {
get: (name: string) => name === "content-type" ? "text/html" : null
},
text: () => Promise.resolve("<html><body>Error page</body></html>")
} as any;
}
return originalFetch(url);
});
const config = {
userId: "user-id",
giteaConfig: {
url: "https://gitea.example.com",
token: "gitea-token"
}
};
try {
await getOrCreateGiteaOrg({
orgName: "test-org",
config
});
// Should not reach here
expect(true).toBe(false);
} catch (error) {
// Should catch the content-type error
expect(error).toBeInstanceOf(Error);
expect((error as Error).message).toContain("Invalid response format from Gitea API");
expect((error as Error).message).toContain("text/html");
} finally {
// Restore original fetch
global.fetch = originalFetch;
}
});
test("mirrorGitHubOrgToGitea handles empty organizations correctly", async () => {
// Mock the createMirrorJob function
const mockCreateMirrorJob = mock(() => Promise.resolve("job-id"));
// Mock the getOrCreateGiteaOrg function
const mockGetOrCreateGiteaOrg = mock(() => Promise.resolve("gitea-org-id"));
// Create a test version of the function with mocked dependencies
const testMirrorGitHubOrgToGitea = async ({
organization,
config,
}: {
organization: any;
config: any;
}) => {
// Simulate the function logic for empty organization
console.log(`Mirroring organization ${organization.name}`);
// Mock: get or create Gitea org
await mockGetOrCreateGiteaOrg();
// Mock: query the db with the org name and get the repos
const orgRepos: any[] = []; // Empty array to simulate no repositories
if (orgRepos.length === 0) {
console.log(`No repositories found for organization ${organization.name} - marking as successfully mirrored`);
} else {
console.log(`Mirroring ${orgRepos.length} repositories for organization ${organization.name}`);
// Repository processing would happen here
}
console.log(`Organization ${organization.name} mirrored successfully`);
// Mock: Append log for "mirrored" status
await mockCreateMirrorJob({
userId: config.userId,
organizationId: organization.id,
organizationName: organization.name,
message: `Successfully mirrored organization: ${organization.name}`,
details: orgRepos.length === 0
? `Organization ${organization.name} was processed successfully (no repositories found).`
: `Organization ${organization.name} was mirrored to Gitea with ${orgRepos.length} repositories.`,
status: "mirrored",
});
};
// Create mock organization
const organization = {
id: "org-id",
name: "empty-org",
status: "imported"
};
// Create mock config
const config = {
id: "config-id",
userId: "user-id",
githubConfig: {
token: "github-token"
},
giteaConfig: {
url: "https://gitea.example.com",
token: "gitea-token"
}
};
// Call the test function
await testMirrorGitHubOrgToGitea({
organization,
config
});
// Verify that the mirror job was created with the correct details for empty org
expect(mockCreateMirrorJob).toHaveBeenCalledWith({
userId: "user-id",
organizationId: "org-id",
organizationName: "empty-org",
message: "Successfully mirrored organization: empty-org",
details: "Organization empty-org was processed successfully (no repositories found).",
status: "mirrored",
});
// Verify that getOrCreateGiteaOrg was called
expect(mockGetOrCreateGiteaOrg).toHaveBeenCalled();
});
});

View File

@@ -6,7 +6,7 @@ import {
import { Octokit } from "@octokit/rest";
import type { Config } from "@/types/config";
import type { Organization, Repository } from "./db/schema";
import superagent from "superagent";
import { httpPost, httpGet } from "./http-client";
import { createMirrorJob } from "./helpers";
import { db, organizations, repositories } from "./db";
import { eq } from "drizzle-orm";
@@ -181,19 +181,17 @@ export const mirrorGithubRepoToGitea = async ({
const apiUrl = `${config.giteaConfig.url}/api/v1/repos/migrate`;
const response = await superagent
.post(apiUrl)
.set("Authorization", `token ${config.giteaConfig.token}`)
.set("Content-Type", "application/json")
.send({
clone_addr: cloneAddress,
repo_name: repository.name,
mirror: true,
private: repository.isPrivate,
repo_owner: config.giteaConfig.username,
description: "",
service: "git",
});
const response = await httpPost(apiUrl, {
clone_addr: cloneAddress,
repo_name: repository.name,
mirror: true,
private: repository.isPrivate,
repo_owner: config.giteaConfig.username,
description: "",
service: "git",
}, {
"Authorization": `token ${config.giteaConfig.token}`,
});
// clone issues
if (config.githubConfig.mirrorIssues) {
@@ -229,7 +227,7 @@ export const mirrorGithubRepoToGitea = async ({
status: "mirrored",
});
return response.body;
return response.data;
} catch (error) {
console.error(
`Error while mirroring repository ${repository.name}: ${
@@ -283,6 +281,8 @@ export async function getOrCreateGiteaOrg({
}
try {
console.log(`Attempting to get or create Gitea organization: ${orgName}`);
const orgRes = await fetch(
`${config.giteaConfig.url}/api/v1/orgs/${orgName}`,
{
@@ -293,13 +293,36 @@ export async function getOrCreateGiteaOrg({
}
);
console.log(`Get org response status: ${orgRes.status} for org: ${orgName}`);
if (orgRes.ok) {
const org = await orgRes.json();
// Note: Organization events are handled by the main mirroring process
// to avoid duplicate events
return org.id;
// Check if response is actually JSON
const contentType = orgRes.headers.get("content-type");
if (!contentType || !contentType.includes("application/json")) {
console.warn(`Expected JSON response but got content-type: ${contentType}`);
const responseText = await orgRes.text();
console.warn(`Response body: ${responseText}`);
throw new Error(`Invalid response format from Gitea API. Expected JSON but got: ${contentType}`);
}
// Clone the response to handle potential JSON parsing errors
const orgResClone = orgRes.clone();
try {
const org = await orgRes.json();
console.log(`Successfully retrieved existing org: ${orgName} with ID: ${org.id}`);
// Note: Organization events are handled by the main mirroring process
// to avoid duplicate events
return org.id;
} catch (jsonError) {
const responseText = await orgResClone.text();
console.error(`Failed to parse JSON response for existing org: ${responseText}`);
throw new Error(`Failed to parse JSON response from Gitea API: ${jsonError instanceof Error ? jsonError.message : String(jsonError)}`);
}
}
console.log(`Organization ${orgName} not found, attempting to create it`);
const createRes = await fetch(`${config.giteaConfig.url}/api/v1/orgs`, {
method: "POST",
headers: {
@@ -314,21 +337,46 @@ export async function getOrCreateGiteaOrg({
}),
});
console.log(`Create org response status: ${createRes.status} for org: ${orgName}`);
if (!createRes.ok) {
throw new Error(`Failed to create Gitea org: ${await createRes.text()}`);
const errorText = await createRes.text();
console.error(`Failed to create org ${orgName}. Status: ${createRes.status}, Response: ${errorText}`);
throw new Error(`Failed to create Gitea org: ${errorText}`);
}
// Check if response is actually JSON
const createContentType = createRes.headers.get("content-type");
if (!createContentType || !createContentType.includes("application/json")) {
console.warn(`Expected JSON response but got content-type: ${createContentType}`);
const responseText = await createRes.text();
console.warn(`Response body: ${responseText}`);
throw new Error(`Invalid response format from Gitea API. Expected JSON but got: ${createContentType}`);
}
// Note: Organization creation events are handled by the main mirroring process
// to avoid duplicate events
const newOrg = await createRes.json();
return newOrg.id;
// Clone the response to handle potential JSON parsing errors
const createResClone = createRes.clone();
try {
const newOrg = await createRes.json();
console.log(`Successfully created new org: ${orgName} with ID: ${newOrg.id}`);
return newOrg.id;
} catch (jsonError) {
const responseText = await createResClone.text();
console.error(`Failed to parse JSON response for new org: ${responseText}`);
throw new Error(`Failed to parse JSON response from Gitea API: ${jsonError instanceof Error ? jsonError.message : String(jsonError)}`);
}
} catch (error) {
const errorMessage =
error instanceof Error
? error.message
: "Unknown error occurred in getOrCreateGiteaOrg.";
console.error(`Error in getOrCreateGiteaOrg for ${orgName}: ${errorMessage}`);
await createMirrorJob({
userId: config.userId,
organizationId: orgId,
@@ -410,17 +458,15 @@ export async function mirrorGitHubRepoToGiteaOrg({
const apiUrl = `${config.giteaConfig.url}/api/v1/repos/migrate`;
const migrateRes = await superagent
.post(apiUrl)
.set("Authorization", `token ${config.giteaConfig.token}`)
.set("Content-Type", "application/json")
.send({
clone_addr: cloneAddress,
uid: giteaOrgId,
repo_name: repository.name,
mirror: true,
private: repository.isPrivate,
});
const migrateRes = await httpPost(apiUrl, {
clone_addr: cloneAddress,
uid: giteaOrgId,
repo_name: repository.name,
mirror: true,
private: repository.isPrivate,
}, {
"Authorization": `token ${config.giteaConfig.token}`,
});
// Clone issues
if (config.githubConfig?.mirrorIssues) {
@@ -458,7 +504,7 @@ export async function mirrorGitHubRepoToGiteaOrg({
status: "mirrored",
});
return migrateRes.body;
return migrateRes.data;
} catch (error) {
console.error(
`Error while mirroring repository ${repository.name}: ${
@@ -583,60 +629,59 @@ export async function mirrorGitHubOrgToGitea({
.where(eq(repositories.organization, organization.name));
if (orgRepos.length === 0) {
console.log(`No repositories found for organization ${organization.name}`);
return;
}
console.log(`No repositories found for organization ${organization.name} - marking as successfully mirrored`);
} else {
console.log(`Mirroring ${orgRepos.length} repositories for organization ${organization.name}`);
console.log(`Mirroring ${orgRepos.length} repositories for organization ${organization.name}`);
// Import the processWithRetry function
const { processWithRetry } = await import("@/lib/utils/concurrency");
// Import the processWithRetry function
const { processWithRetry } = await import("@/lib/utils/concurrency");
// Process repositories in parallel with concurrency control
await processWithRetry(
orgRepos,
async (repo) => {
// Prepare repository data
const repoData = {
...repo,
status: repo.status as RepoStatus,
visibility: repo.visibility as RepositoryVisibility,
lastMirrored: repo.lastMirrored ?? undefined,
errorMessage: repo.errorMessage ?? undefined,
organization: repo.organization ?? undefined,
forkedFrom: repo.forkedFrom ?? undefined,
mirroredLocation: repo.mirroredLocation || "",
};
// Process repositories in parallel with concurrency control
await processWithRetry(
orgRepos,
async (repo) => {
// Prepare repository data
const repoData = {
...repo,
status: repo.status as RepoStatus,
visibility: repo.visibility as RepositoryVisibility,
lastMirrored: repo.lastMirrored ?? undefined,
errorMessage: repo.errorMessage ?? undefined,
organization: repo.organization ?? undefined,
forkedFrom: repo.forkedFrom ?? undefined,
mirroredLocation: repo.mirroredLocation || "",
};
// Log the start of mirroring
console.log(`Starting mirror for repository: ${repo.name} in organization ${organization.name}`);
// Log the start of mirroring
console.log(`Starting mirror for repository: ${repo.name} in organization ${organization.name}`);
// Mirror the repository
await mirrorGitHubRepoToGiteaOrg({
octokit,
config,
repository: repoData,
giteaOrgId,
orgName: organization.name,
});
// Mirror the repository
await mirrorGitHubRepoToGiteaOrg({
octokit,
config,
repository: repoData,
giteaOrgId,
orgName: organization.name,
});
return repo;
},
{
concurrencyLimit: 3, // Process 3 repositories at a time
maxRetries: 2,
retryDelay: 2000,
onProgress: (completed, total, result) => {
const percentComplete = Math.round((completed / total) * 100);
if (result) {
console.log(`Mirrored repository "${result.name}" in organization ${organization.name} (${completed}/${total}, ${percentComplete}%)`);
}
return repo;
},
onRetry: (repo, error, attempt) => {
console.log(`Retrying repository ${repo.name} in organization ${organization.name} (attempt ${attempt}): ${error.message}`);
{
concurrencyLimit: 3, // Process 3 repositories at a time
maxRetries: 2,
retryDelay: 2000,
onProgress: (completed, total, result) => {
const percentComplete = Math.round((completed / total) * 100);
if (result) {
console.log(`Mirrored repository "${result.name}" in organization ${organization.name} (${completed}/${total}, ${percentComplete}%)`);
}
},
onRetry: (repo, error, attempt) => {
console.log(`Retrying repository ${repo.name} in organization ${organization.name} (attempt ${attempt}): ${error.message}`);
}
}
}
);
);
}
console.log(`Organization ${organization.name} mirrored successfully`);
@@ -657,7 +702,9 @@ export async function mirrorGitHubOrgToGitea({
organizationId: organization.id,
organizationName: organization.name,
message: `Successfully mirrored organization: ${organization.name}`,
details: `Organization ${organization.name} was mirrored to Gitea.`,
details: orgRepos.length === 0
? `Organization ${organization.name} was processed successfully (no repositories found).`
: `Organization ${organization.name} was mirrored to Gitea with ${orgRepos.length} repositories.`,
status: repoStatusEnum.parse("mirrored"),
});
} catch (error) {
@@ -751,9 +798,9 @@ export const syncGiteaRepo = async ({
// Use the actual owner where the repo was found
const apiUrl = `${config.giteaConfig.url}/api/v1/repos/${actualOwner}/${repository.name}/mirror-sync`;
const response = await superagent
.post(apiUrl)
.set("Authorization", `token ${config.giteaConfig.token}`);
const response = await httpPost(apiUrl, undefined, {
"Authorization": `token ${config.giteaConfig.token}`,
});
// Mark repo as "synced" in DB
await db
@@ -779,7 +826,7 @@ export const syncGiteaRepo = async ({
console.log(`Repository ${repository.name} synced successfully`);
return response.body;
return response.data;
} catch (error) {
console.error(
`Error while syncing repository ${repository.name}: ${
@@ -866,13 +913,14 @@ export const mirrorGitRepoIssuesToGitea = async ({
}
// Get existing labels from Gitea
const giteaLabelsRes = await superagent
.get(
`${config.giteaConfig.url}/api/v1/repos/${repoOrigin}/${repository.name}/labels`
)
.set("Authorization", `token ${config.giteaConfig.token}`);
const giteaLabelsRes = await httpGet(
`${config.giteaConfig.url}/api/v1/repos/${repoOrigin}/${repository.name}/labels`,
{
"Authorization": `token ${config.giteaConfig.token}`,
}
);
const giteaLabels = giteaLabelsRes.body;
const giteaLabels = giteaLabelsRes.data;
const labelMap = new Map<string, number>(
giteaLabels.map((label: any) => [label.name, label.id])
);
@@ -897,15 +945,16 @@ export const mirrorGitRepoIssuesToGitea = async ({
giteaLabelIds.push(labelMap.get(name)!);
} else {
try {
const created = await superagent
.post(
`${config.giteaConfig.url}/api/v1/repos/${repoOrigin}/${repository.name}/labels`
)
.set("Authorization", `token ${config.giteaConfig.token}`)
.send({ name, color: "#ededed" }); // Default color
const created = await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${repoOrigin}/${repository.name}/labels`,
{ name, color: "#ededed" }, // Default color
{
"Authorization": `token ${config.giteaConfig!.token}`,
}
);
labelMap.set(name, created.body.id);
giteaLabelIds.push(created.body.id);
labelMap.set(name, created.data.id);
giteaLabelIds.push(created.data.id);
} catch (labelErr) {
console.error(
`Failed to create label "${name}" in Gitea: ${labelErr}`
@@ -931,12 +980,13 @@ export const mirrorGitRepoIssuesToGitea = async ({
};
// Create the issue in Gitea
const createdIssue = await superagent
.post(
`${config.giteaConfig.url}/api/v1/repos/${repoOrigin}/${repository.name}/issues`
)
.set("Authorization", `token ${config.giteaConfig.token}`)
.send(issuePayload);
const createdIssue = await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${repoOrigin}/${repository.name}/issues`,
issuePayload,
{
"Authorization": `token ${config.giteaConfig!.token}`,
}
);
// Clone comments
const comments = await octokit.paginate(
@@ -955,21 +1005,22 @@ export const mirrorGitRepoIssuesToGitea = async ({
await processWithRetry(
comments,
async (comment) => {
await superagent
.post(
`${config.giteaConfig.url}/api/v1/repos/${repoOrigin}/${repository.name}/issues/${createdIssue.body.number}/comments`
)
.set("Authorization", `token ${config.giteaConfig.token}`)
.send({
await httpPost(
`${config.giteaConfig!.url}/api/v1/repos/${repoOrigin}/${repository.name}/issues/${createdIssue.data.number}/comments`,
{
body: `@${comment.user?.login} commented on GitHub:\n\n${comment.body}`,
});
},
{
"Authorization": `token ${config.giteaConfig!.token}`,
}
);
return comment;
},
{
concurrencyLimit: 5,
maxRetries: 2,
retryDelay: 1000,
onRetry: (comment, error, attempt) => {
onRetry: (_comment, error, attempt) => {
console.log(`Retrying comment (attempt ${attempt}): ${error.message}`);
}
}

204
src/lib/http-client.ts Normal file
View File

@@ -0,0 +1,204 @@
/**
* HTTP client utility functions using fetch() for consistent error handling
*/
export interface HttpResponse<T = any> {
data: T;
status: number;
statusText: string;
headers: Headers;
}
export class HttpError extends Error {
constructor(
message: string,
public status: number,
public statusText: string,
public response?: string
) {
super(message);
this.name = 'HttpError';
}
}
/**
* Enhanced fetch with consistent error handling and JSON parsing
*/
export async function httpRequest<T = any>(
url: string,
options: RequestInit = {}
): Promise<HttpResponse<T>> {
try {
const response = await fetch(url, {
...options,
headers: {
'Content-Type': 'application/json',
...options.headers,
},
});
// Clone response for error handling
const responseClone = response.clone();
if (!response.ok) {
let errorMessage = `HTTP ${response.status}: ${response.statusText}`;
let responseText = '';
try {
responseText = await responseClone.text();
if (responseText) {
errorMessage += ` - ${responseText}`;
}
} catch {
// Ignore text parsing errors
}
throw new HttpError(
errorMessage,
response.status,
response.statusText,
responseText
);
}
// Check content type for JSON responses
const contentType = response.headers.get('content-type');
let data: T;
if (contentType && contentType.includes('application/json')) {
try {
data = await response.json();
} catch (jsonError) {
const responseText = await responseClone.text();
// Enhanced JSON parsing error logging
console.error("=== JSON PARSING ERROR ===");
console.error("URL:", url);
console.error("Status:", response.status, response.statusText);
console.error("Content-Type:", contentType);
console.error("Response length:", responseText.length);
console.error("Response preview (first 500 chars):", responseText.substring(0, 500));
console.error("JSON Error:", jsonError instanceof Error ? jsonError.message : String(jsonError));
console.error("========================");
throw new HttpError(
`Failed to parse JSON response from ${url}: ${jsonError instanceof Error ? jsonError.message : String(jsonError)}. Response: ${responseText.substring(0, 200)}${responseText.length > 200 ? '...' : ''}`,
response.status,
response.statusText,
responseText
);
}
} else {
// For non-JSON responses, return text as data
data = (await response.text()) as unknown as T;
}
return {
data,
status: response.status,
statusText: response.statusText,
headers: response.headers,
};
} catch (error) {
if (error instanceof HttpError) {
throw error;
}
// Handle network errors, etc.
throw new HttpError(
`Network error: ${error instanceof Error ? error.message : String(error)}`,
0,
'Network Error'
);
}
}
/**
* GET request
*/
export async function httpGet<T = any>(
url: string,
headers?: Record<string, string>
): Promise<HttpResponse<T>> {
return httpRequest<T>(url, {
method: 'GET',
headers,
});
}
/**
* POST request
*/
export async function httpPost<T = any>(
url: string,
body?: any,
headers?: Record<string, string>
): Promise<HttpResponse<T>> {
return httpRequest<T>(url, {
method: 'POST',
headers,
body: body ? JSON.stringify(body) : undefined,
});
}
/**
* PUT request
*/
export async function httpPut<T = any>(
url: string,
body?: any,
headers?: Record<string, string>
): Promise<HttpResponse<T>> {
return httpRequest<T>(url, {
method: 'PUT',
headers,
body: body ? JSON.stringify(body) : undefined,
});
}
/**
* DELETE request
*/
export async function httpDelete<T = any>(
url: string,
headers?: Record<string, string>
): Promise<HttpResponse<T>> {
return httpRequest<T>(url, {
method: 'DELETE',
headers,
});
}
/**
* Gitea-specific HTTP client with authentication
*/
export class GiteaHttpClient {
constructor(
private baseUrl: string,
private token: string
) {}
private getHeaders(additionalHeaders?: Record<string, string>): Record<string, string> {
return {
'Authorization': `token ${this.token}`,
'Content-Type': 'application/json',
...additionalHeaders,
};
}
async get<T = any>(endpoint: string): Promise<HttpResponse<T>> {
return httpGet<T>(`${this.baseUrl}${endpoint}`, this.getHeaders());
}
async post<T = any>(endpoint: string, body?: any): Promise<HttpResponse<T>> {
return httpPost<T>(`${this.baseUrl}${endpoint}`, body, this.getHeaders());
}
async put<T = any>(endpoint: string, body?: any): Promise<HttpResponse<T>> {
return httpPut<T>(`${this.baseUrl}${endpoint}`, body, this.getHeaders());
}
async delete<T = any>(endpoint: string): Promise<HttpResponse<T>> {
return httpDelete<T>(`${this.baseUrl}${endpoint}`, this.getHeaders());
}
}

View File

@@ -1,35 +1,35 @@
import { describe, test, expect } from "bun:test";
import { jsonResponse, formatDate, truncate, safeParse } from "./utils";
import { jsonResponse, formatDate, truncate, safeParse, parseErrorMessage, showErrorToast } from "./utils";
describe("jsonResponse", () => {
test("creates a Response with JSON content", () => {
const data = { message: "Hello, world!" };
const response = jsonResponse({ data });
expect(response).toBeInstanceOf(Response);
expect(response.status).toBe(200);
expect(response.headers.get("Content-Type")).toBe("application/json");
});
test("uses the provided status code", () => {
const data = { error: "Not found" };
const response = jsonResponse({ data, status: 404 });
expect(response.status).toBe(404);
});
test("correctly serializes complex objects", async () => {
const now = new Date();
const data = {
const data = {
message: "Complex object",
date: now,
nested: { foo: "bar" },
array: [1, 2, 3]
};
const response = jsonResponse({ data });
const responseBody = await response.json();
expect(responseBody).toEqual({
message: "Complex object",
date: now.toISOString(),
@@ -43,22 +43,22 @@ describe("formatDate", () => {
test("formats a date object", () => {
const date = new Date("2023-01-15T12:30:45Z");
const formatted = formatDate(date);
// The exact format might depend on the locale, so we'll check for parts
expect(formatted).toContain("2023");
expect(formatted).toContain("January");
expect(formatted).toContain("15");
});
test("formats a date string", () => {
const dateStr = "2023-01-15T12:30:45Z";
const formatted = formatDate(dateStr);
expect(formatted).toContain("2023");
expect(formatted).toContain("January");
expect(formatted).toContain("15");
});
test("returns 'Never' for null or undefined", () => {
expect(formatDate(null)).toBe("Never");
expect(formatDate(undefined)).toBe("Never");
@@ -69,18 +69,18 @@ describe("truncate", () => {
test("truncates a string that exceeds the length", () => {
const str = "This is a long string that needs truncation";
const truncated = truncate(str, 10);
expect(truncated).toBe("This is a ...");
expect(truncated.length).toBe(13); // 10 chars + "..."
});
test("does not truncate a string that is shorter than the length", () => {
const str = "Short";
const truncated = truncate(str, 10);
expect(truncated).toBe("Short");
});
test("handles empty strings", () => {
expect(truncate("", 10)).toBe("");
});
@@ -90,21 +90,71 @@ describe("safeParse", () => {
test("parses valid JSON strings", () => {
const jsonStr = '{"name":"John","age":30}';
const parsed = safeParse(jsonStr);
expect(parsed).toEqual({ name: "John", age: 30 });
});
test("returns undefined for invalid JSON strings", () => {
const invalidJson = '{"name":"John",age:30}'; // Missing quotes around age
const parsed = safeParse(invalidJson);
expect(parsed).toBeUndefined();
});
test("returns the original value for non-string inputs", () => {
const obj = { name: "John", age: 30 };
const parsed = safeParse(obj);
expect(parsed).toBe(obj);
});
});
describe("parseErrorMessage", () => {
test("parses JSON error with error and troubleshooting fields", () => {
const errorMessage = JSON.stringify({
error: "Unexpected end of JSON input",
errorType: "SyntaxError",
timestamp: "2025-05-28T09:08:02.37Z",
troubleshooting: "JSON parsing error detected. Check Gitea server status and logs. Ensure Gitea is returning valid JSON responses."
});
const result = parseErrorMessage(errorMessage);
expect(result.title).toBe("Unexpected end of JSON input");
expect(result.description).toBe("JSON parsing error detected. Check Gitea server status and logs. Ensure Gitea is returning valid JSON responses.");
expect(result.isStructured).toBe(true);
});
test("parses JSON error with title and description fields", () => {
const errorMessage = JSON.stringify({
title: "Connection Failed",
description: "Unable to connect to the server. Please check your network connection."
});
const result = parseErrorMessage(errorMessage);
expect(result.title).toBe("Connection Failed");
expect(result.description).toBe("Unable to connect to the server. Please check your network connection.");
expect(result.isStructured).toBe(true);
});
test("handles plain string error messages", () => {
const errorMessage = "Simple error message";
const result = parseErrorMessage(errorMessage);
expect(result.title).toBe("Simple error message");
expect(result.description).toBeUndefined();
expect(result.isStructured).toBe(false);
});
test("handles Error objects", () => {
const error = new Error("Something went wrong");
const result = parseErrorMessage(error);
expect(result.title).toBe("Something went wrong");
expect(result.description).toBeUndefined();
expect(result.isStructured).toBe(false);
});
});

View File

@@ -1,7 +1,6 @@
import { clsx, type ClassValue } from "clsx";
import { twMerge } from "tailwind-merge";
import axios from "axios";
import type { AxiosError, AxiosRequestConfig } from "axios";
import { httpRequest, HttpError } from "@/lib/http-client";
import type { RepoStatus } from "@/types/Repository";
export const API_BASE = "/api";
@@ -37,27 +36,148 @@ export function safeParse<T>(value: unknown): T | undefined {
return value as T;
}
// Enhanced error message parsing for toast notifications
export interface ParsedErrorMessage {
title: string;
description?: string;
isStructured: boolean;
}
export function parseErrorMessage(error: unknown): ParsedErrorMessage {
// Handle Error objects
if (error instanceof Error) {
return parseErrorMessage(error.message);
}
// Handle string messages
if (typeof error === "string") {
// Try to parse as JSON first
try {
const parsed = JSON.parse(error);
// Check for common structured error formats
if (typeof parsed === "object" && parsed !== null) {
// Format 1: { error: "message", errorType: "type", troubleshooting: "info" }
if (parsed.error) {
return {
title: parsed.error,
description: parsed.troubleshooting || parsed.errorType || undefined,
isStructured: true,
};
}
// Format 2: { title: "title", description: "desc" }
if (parsed.title) {
return {
title: parsed.title,
description: parsed.description || undefined,
isStructured: true,
};
}
// Format 3: { message: "msg", details: "details" }
if (parsed.message) {
return {
title: parsed.message,
description: parsed.details || undefined,
isStructured: true,
};
}
}
} catch {
// Not valid JSON, treat as plain string
}
// Plain string message
return {
title: error,
description: undefined,
isStructured: false,
};
}
// Handle objects directly
if (typeof error === "object" && error !== null) {
const errorObj = error as any;
if (errorObj.error) {
return {
title: errorObj.error,
description: errorObj.troubleshooting || errorObj.errorType || undefined,
isStructured: true,
};
}
if (errorObj.title) {
return {
title: errorObj.title,
description: errorObj.description || undefined,
isStructured: true,
};
}
if (errorObj.message) {
return {
title: errorObj.message,
description: errorObj.details || undefined,
isStructured: true,
};
}
}
// Fallback for unknown types
return {
title: String(error),
description: undefined,
isStructured: false,
};
}
// Enhanced toast helper that parses structured error messages
export function showErrorToast(error: unknown, toast: any) {
const parsed = parseErrorMessage(error);
if (parsed.description) {
// Use sonner's rich toast format with title and description
toast.error(parsed.title, {
description: parsed.description,
});
} else {
// Simple error toast
toast.error(parsed.title);
}
}
// Helper function for API requests
export async function apiRequest<T>(
endpoint: string,
options: AxiosRequestConfig = {}
options: (RequestInit & { data?: any }) = {}
): Promise<T> {
try {
const response = await axios<T>(`${API_BASE}${endpoint}`, {
// Handle the custom 'data' property by converting it to 'body'
const { data, ...requestOptions } = options;
const finalOptions: RequestInit = {
headers: {
"Content-Type": "application/json",
...(options.headers || {}),
...(requestOptions.headers || {}),
},
...options,
});
...requestOptions,
};
// If data is provided, stringify it and set as body
if (data !== undefined) {
finalOptions.body = JSON.stringify(data);
}
const response = await httpRequest<T>(`${API_BASE}${endpoint}`, finalOptions);
return response.data;
} catch (err) {
const error = err as AxiosError<{ message?: string }>;
const error = err as HttpError;
const message =
error.response?.data?.message ||
error.response ||
error.message ||
"An unknown error occurred";

View File

@@ -5,19 +5,19 @@ describe("processInParallel", () => {
test("processes items in parallel with concurrency control", async () => {
// Create an array of numbers to process
const items = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10];
// Create a mock function to track execution
const processItem = mock(async (item: number) => {
// Simulate async work
await new Promise(resolve => setTimeout(resolve, 10));
return item * 2;
});
// Create a mock progress callback
const onProgress = mock((completed: number, total: number, result?: number) => {
// Progress tracking
});
// Process the items with a concurrency limit of 3
const results = await processInParallel(
items,
@@ -25,25 +25,25 @@ describe("processInParallel", () => {
3,
onProgress
);
// Verify results
expect(results).toEqual([2, 4, 6, 8, 10, 12, 14, 16, 18, 20]);
// Verify that processItem was called for each item
expect(processItem).toHaveBeenCalledTimes(10);
// Verify that onProgress was called for each item
expect(onProgress).toHaveBeenCalledTimes(10);
// Verify the last call to onProgress had the correct completed/total values
expect(onProgress.mock.calls[9][0]).toBe(10); // completed
expect(onProgress.mock.calls[9][1]).toBe(10); // total
});
test("handles errors in processing", async () => {
// Create an array of numbers to process
const items = [1, 2, 3, 4, 5];
// Create a mock function that throws an error for item 3
const processItem = mock(async (item: number) => {
if (item === 3) {
@@ -51,24 +51,24 @@ describe("processInParallel", () => {
}
return item * 2;
});
// Create a spy for console.error
const originalConsoleError = console.error;
const consoleErrorMock = mock(() => {});
console.error = consoleErrorMock;
try {
// Process the items
const results = await processInParallel(items, processItem);
// Verify results (should have 4 items, missing the one that errored)
expect(results).toEqual([2, 4, 8, 10]);
// Verify that processItem was called for each item
expect(processItem).toHaveBeenCalledTimes(5);
// Verify that console.error was called once
expect(consoleErrorMock).toHaveBeenCalledTimes(1);
// Verify that console.error was called (enhanced logging calls it multiple times)
expect(consoleErrorMock).toHaveBeenCalled();
} finally {
// Restore console.error
console.error = originalConsoleError;
@@ -80,51 +80,51 @@ describe("processWithRetry", () => {
test("retries failed operations", async () => {
// Create an array of numbers to process
const items = [1, 2, 3];
// Create a counter to track retry attempts
const attemptCounts: Record<number, number> = { 1: 0, 2: 0, 3: 0 };
// Create a mock function that fails on first attempt for item 2
const processItem = mock(async (item: number) => {
attemptCounts[item]++;
if (item === 2 && attemptCounts[item] === 1) {
throw new Error("Temporary error");
}
return item * 2;
});
// Create a mock for the onRetry callback
const onRetry = mock((item: number, error: Error, attempt: number) => {
// Retry tracking
});
// Process the items with retry
const results = await processWithRetry(items, processItem, {
maxRetries: 2,
retryDelay: 10,
onRetry,
});
// Verify results
expect(results).toEqual([2, 4, 6]);
// Verify that item 2 was retried once
expect(attemptCounts[1]).toBe(1); // No retries
expect(attemptCounts[2]).toBe(2); // One retry
expect(attemptCounts[3]).toBe(1); // No retries
// Verify that onRetry was called once
expect(onRetry).toHaveBeenCalledTimes(1);
expect(onRetry.mock.calls[0][0]).toBe(2); // item
expect(onRetry.mock.calls[0][2]).toBe(1); // attempt
});
test("gives up after max retries", async () => {
// Create an array of numbers to process
const items = [1, 2];
// Create a mock function that always fails for item 2
const processItem = mock(async (item: number) => {
if (item === 2) {
@@ -132,17 +132,17 @@ describe("processWithRetry", () => {
}
return item * 2;
});
// Create a mock for the onRetry callback
const onRetry = mock((item: number, error: Error, attempt: number) => {
// Retry tracking
});
// Create a spy for console.error
const originalConsoleError = console.error;
const consoleErrorMock = mock(() => {});
console.error = consoleErrorMock;
try {
// Process the items with retry
const results = await processWithRetry(items, processItem, {
@@ -150,15 +150,15 @@ describe("processWithRetry", () => {
retryDelay: 10,
onRetry,
});
// Verify results (should have 1 item, missing the one that errored)
expect(results).toEqual([2]);
// Verify that onRetry was called twice (for 2 retry attempts)
expect(onRetry).toHaveBeenCalledTimes(2);
// Verify that console.error was called once
expect(consoleErrorMock).toHaveBeenCalledTimes(1);
// Verify that console.error was called (enhanced logging calls it multiple times)
expect(consoleErrorMock).toHaveBeenCalled();
} finally {
// Restore console.error
console.error = originalConsoleError;

View File

@@ -46,11 +46,25 @@ export async function processInParallel<T, R>(
const batchResults = await Promise.allSettled(batchPromises);
// Process results and handle errors
for (const result of batchResults) {
for (let j = 0; j < batchResults.length; j++) {
const result = batchResults[j];
if (result.status === 'fulfilled') {
results.push(result.value);
} else {
console.error('Error processing item:', result.reason);
const itemIndex = i + j;
console.error("=== BATCH ITEM PROCESSING ERROR ===");
console.error("Batch index:", Math.floor(i / concurrencyLimit));
console.error("Item index in batch:", j);
console.error("Global item index:", itemIndex);
console.error("Error type:", result.reason?.constructor?.name);
console.error("Error message:", result.reason instanceof Error ? result.reason.message : String(result.reason));
if (result.reason instanceof Error && result.reason.message.includes('JSON')) {
console.error("🚨 JSON parsing error in batch processing");
console.error("This indicates an API response issue from Gitea");
}
console.error("==================================");
}
}
}
@@ -139,6 +153,21 @@ export async function processWithRetry<T, R>(
const delay = retryDelay * Math.pow(2, attempt - 1);
await new Promise(resolve => setTimeout(resolve, delay));
} else {
// Enhanced error logging for final failure
console.error("=== ITEM PROCESSING FAILED (MAX RETRIES EXCEEDED) ===");
console.error("Item:", getItemId ? getItemId(item) : 'unknown');
console.error("Error type:", lastError.constructor.name);
console.error("Error message:", lastError.message);
console.error("Attempts made:", maxRetries + 1);
if (lastError.message.includes('JSON')) {
console.error("🚨 JSON-related error detected in item processing");
console.error("This suggests an issue with API responses from Gitea");
}
console.error("Stack trace:", lastError.stack);
console.error("================================================");
throw lastError;
}
}

View File

@@ -1,5 +1,4 @@
import { describe, test, expect, mock, beforeEach, afterEach } from "bun:test";
import axios from "axios";
// Mock the POST function
const mockPOST = mock(async ({ request }) => {

View File

@@ -1,5 +1,5 @@
import type { APIRoute } from 'astro';
import axios from 'axios';
import { httpGet, HttpError } from '@/lib/http-client';
export const POST: APIRoute = async ({ request }) => {
try {
@@ -25,11 +25,9 @@ export const POST: APIRoute = async ({ request }) => {
const baseUrl = url.endsWith('/') ? url.slice(0, -1) : url;
// Test the connection by fetching the authenticated user
const response = await axios.get(`${baseUrl}/api/v1/user`, {
headers: {
'Authorization': `token ${token}`,
'Accept': 'application/json',
},
const response = await httpGet(`${baseUrl}/api/v1/user`, {
'Authorization': `token ${token}`,
'Accept': 'application/json',
});
const data = response.data;
@@ -72,8 +70,8 @@ export const POST: APIRoute = async ({ request }) => {
console.error('Gitea connection test failed:', error);
// Handle specific error types
if (axios.isAxiosError(error) && error.response) {
if (error.response.status === 401) {
if (error instanceof HttpError) {
if (error.status === 401) {
return new Response(
JSON.stringify({
success: false,
@@ -86,7 +84,7 @@ export const POST: APIRoute = async ({ request }) => {
},
}
);
} else if (error.response.status === 404) {
} else if (error.status === 404) {
return new Response(
JSON.stringify({
success: false,
@@ -99,25 +97,23 @@ export const POST: APIRoute = async ({ request }) => {
},
}
);
} else if (error.status === 0) {
// Network error
return new Response(
JSON.stringify({
success: false,
message: 'Could not connect to Gitea server. Please check the URL.',
}),
{
status: 500,
headers: {
'Content-Type': 'application/json',
},
}
);
}
}
// Handle connection errors
if (axios.isAxiosError(error) && (error.code === 'ECONNREFUSED' || error.code === 'ENOTFOUND')) {
return new Response(
JSON.stringify({
success: false,
message: 'Could not connect to Gitea server. Please check the URL.',
}),
{
status: 500,
headers: {
'Content-Type': 'application/json',
},
}
);
}
// Generic error response
return new Response(
JSON.stringify({

View File

@@ -1,7 +1,7 @@
import type { APIRoute } from "astro";
import { db } from "@/lib/db";
import { organizations } from "@/lib/db";
import { eq, sql } from "drizzle-orm";
import { organizations, repositories, configs } from "@/lib/db";
import { eq, sql, and, count } from "drizzle-orm";
import {
membershipRoleEnum,
type OrganizationsApiResponse,
@@ -25,24 +25,114 @@ export const GET: APIRoute = async ({ request }) => {
}
try {
// Fetch the user's active configuration to respect filtering settings
const [config] = await db
.select()
.from(configs)
.where(and(eq(configs.userId, userId), eq(configs.isActive, true)));
if (!config) {
return jsonResponse({
data: {
success: false,
error: "No active configuration found for this user",
},
status: 404,
});
}
const githubConfig = config.githubConfig as {
mirrorStarred: boolean;
skipForks: boolean;
privateRepositories: boolean;
};
const rawOrgs = await db
.select()
.from(organizations)
.where(eq(organizations.userId, userId))
.orderBy(sql`name COLLATE NOCASE`);
const orgsWithIds: Organization[] = rawOrgs.map((org) => ({
...org,
status: repoStatusEnum.parse(org.status),
membershipRole: membershipRoleEnum.parse(org.membershipRole),
lastMirrored: org.lastMirrored ?? undefined,
errorMessage: org.errorMessage ?? undefined,
}));
// Calculate repository breakdowns for each organization
const orgsWithBreakdown = await Promise.all(
rawOrgs.map(async (org) => {
// Build base conditions for this organization (without private/fork filters)
const baseConditions = [
eq(repositories.userId, userId),
eq(repositories.organization, org.name)
];
if (!githubConfig.mirrorStarred) {
baseConditions.push(eq(repositories.isStarred, false));
}
// Get total count with all user config filters applied
const totalConditions = [...baseConditions];
if (githubConfig.skipForks) {
totalConditions.push(eq(repositories.isForked, false));
}
if (!githubConfig.privateRepositories) {
totalConditions.push(eq(repositories.isPrivate, false));
}
const [totalCount] = await db
.select({ count: count() })
.from(repositories)
.where(and(...totalConditions));
// Get public count
const publicConditions = [...baseConditions, eq(repositories.isPrivate, false)];
if (githubConfig.skipForks) {
publicConditions.push(eq(repositories.isForked, false));
}
const [publicCount] = await db
.select({ count: count() })
.from(repositories)
.where(and(...publicConditions));
// Get private count (only if private repos are enabled in config)
const [privateCount] = githubConfig.privateRepositories ? await db
.select({ count: count() })
.from(repositories)
.where(
and(
...baseConditions,
eq(repositories.isPrivate, true),
...(githubConfig.skipForks ? [eq(repositories.isForked, false)] : [])
)
) : [{ count: 0 }];
// Get fork count (only if forks are enabled in config)
const [forkCount] = !githubConfig.skipForks ? await db
.select({ count: count() })
.from(repositories)
.where(
and(
...baseConditions,
eq(repositories.isForked, true),
...(!githubConfig.privateRepositories ? [eq(repositories.isPrivate, false)] : [])
)
) : [{ count: 0 }];
return {
...org,
status: repoStatusEnum.parse(org.status),
membershipRole: membershipRoleEnum.parse(org.membershipRole),
lastMirrored: org.lastMirrored ?? undefined,
errorMessage: org.errorMessage ?? undefined,
repositoryCount: totalCount.count,
publicRepositoryCount: publicCount.count,
privateRepositoryCount: privateCount.count,
forkRepositoryCount: forkCount.count,
};
})
);
const resPayload: OrganizationsApiResponse = {
success: true,
message: "Organizations fetched successfully",
organizations: orgsWithIds,
organizations: orgsWithBreakdown,
};
return jsonResponse({ data: resPayload, status: 200 });

View File

@@ -4,7 +4,7 @@ import { db } from "@/lib/db";
import { ENV } from "@/lib/config";
import { getRecoveryStatus, hasJobsNeedingRecovery } from "@/lib/recovery";
import os from "os";
import axios from "axios";
import { httpGet } from "@/lib/http-client";
// Track when the server started
const serverStartTime = new Date();
@@ -197,9 +197,9 @@ async function checkLatestVersion(): Promise<string> {
try {
// Fetch the latest release from GitHub
const response = await axios.get(
const response = await httpGet(
'https://api.github.com/repos/arunavo4/gitea-mirror/releases/latest',
{ headers: { 'Accept': 'application/vnd.github.v3+json' } }
{ 'Accept': 'application/vnd.github.v3+json' }
);
// Extract version from tag_name (remove 'v' prefix if present)

View File

@@ -165,11 +165,43 @@ export const POST: APIRoute = async ({ request }) => {
headers: { "Content-Type": "application/json" },
});
} catch (error) {
console.error("Error mirroring repositories:", error);
// Enhanced error logging for better debugging
console.error("=== ERROR MIRRORING REPOSITORIES ===");
console.error("Error type:", error?.constructor?.name);
console.error("Error message:", error instanceof Error ? error.message : String(error));
if (error instanceof Error) {
console.error("Error stack:", error.stack);
}
// Log additional context
console.error("Request details:");
console.error("- URL:", request.url);
console.error("- Method:", request.method);
console.error("- Headers:", Object.fromEntries(request.headers.entries()));
// If it's a JSON parsing error, provide more context
if (error instanceof SyntaxError && error.message.includes('JSON')) {
console.error("🚨 JSON PARSING ERROR DETECTED:");
console.error("This suggests the response from Gitea API is not valid JSON");
console.error("Common causes:");
console.error("- Gitea server returned HTML error page instead of JSON");
console.error("- Network connection interrupted");
console.error("- Gitea server is down or misconfigured");
console.error("- Authentication token is invalid");
console.error("Check your Gitea server logs and configuration");
}
console.error("=====================================");
return new Response(
JSON.stringify({
error:
error instanceof Error ? error.message : "An unknown error occurred",
error: error instanceof Error ? error.message : "An unknown error occurred",
errorType: error?.constructor?.name || "Unknown",
timestamp: new Date().toISOString(),
troubleshooting: error instanceof SyntaxError && error.message.includes('JSON')
? "JSON parsing error detected. Check Gitea server status and logs. Ensure Gitea is returning valid JSON responses."
: "Check application logs for more details"
}),
{ status: 500, headers: { "Content-Type": "application/json" } }
);

View File

@@ -33,6 +33,9 @@ export interface GitOrg {
isIncluded: boolean;
status: RepoStatus;
repositoryCount: number;
publicRepositoryCount?: number;
privateRepositoryCount?: number;
forkRepositoryCount?: number;
createdAt: Date;
updatedAt: Date;
}