diff --git a/mcp-gateway/RFP_SCRAPER_SETUP.md b/mcp-gateway/RFP_SCRAPER_SETUP.md deleted file mode 100644 index c82f9a7..0000000 --- a/mcp-gateway/RFP_SCRAPER_SETUP.md +++ /dev/null @@ -1,370 +0,0 @@ -# RFP Scraper MCP - Setup & Configuration Guide - -## Overview - -The **RFP Scraper MCP** is a custom-built tool that searches for Request For Proposal (RFP) opportunities across multiple freelance platforms. It's designed to find control room design, broadcast studio, and AV systems projects. - -### Current Status - -βœ… **Ready to Deploy** - Currently uses mock/sample data -πŸ”„ **Production Ready Architecture** - Prepared for real API integration -⚠️ **API Credentials Optional** - Add when you want live scraping - ---- - -## What It Does - -The RFP Scraper searches for freelance projects on: - -1. **Upwork** - Large freelance marketplace -2. **Freelancer.com** - Global freelance platform -3. **Toptal** - High-end freelance talent network - -### Available Tools - -``` -rfp_search_projects -β”œβ”€β”€ Search by keywords (e.g., "control room", "broadcast studio") -β”œβ”€β”€ Filter by budget range -β”œβ”€β”€ Sort by date, relevance, budget -└── Get paginated results - -rfp_get_project_details -β”œβ”€β”€ Fetch full project information -β”œβ”€β”€ Client details and ratings -β”œβ”€β”€ Deadline and budget -└── Required skills and experience - -rfp_filter_by_budget -β”œβ”€β”€ Filter projects by price range -β”œβ”€β”€ Show budget breakdown -└── Cost analysis -``` - ---- - -## Deployment - -### Step 1: No Changes Needed for Mock Mode - -The RFP Scraper is **already configured** to return sample data. You can deploy and test it immediately: - -```bash -cd /path/to/mcp-gateway -docker-compose up -d rfpscraper-mcp -``` - -### Step 2: Verify It's Running - -```bash -curl http://10.0.0.25:4444/dashboard/status | jq '.services[] | select(.name=="RFP Scraper")' -``` - -Should show: -```json -{ - "name": "RFP Scraper", - "status": "healthy", - "toolCount": 3, - "responseTime": 45 -} -``` - ---- - -## Using Mock Data (Development/Testing) - -The RFP Scraper returns **realistic sample data** for testing: - -```bash -# Example request -curl -X POST http://10.0.0.25:4444/mcp \ - -H "Authorization: Bearer YOUR_API_KEY" \ - -H "Content-Type: application/json" \ - -d '{ - "jsonrpc": "2.0", - "id": 1, - "method": "rfp_search_projects", - "params": { - "keywords": "broadcast studio", - "budget_min": 5000, - "budget_max": 50000 - } - }' -``` - -Sample response: -```json -{ - "projects": [ - { - "id": "upwork_12345", - "platform": "upwork", - "title": "Professional Broadcast Studio Setup for Sports Network", - "budget": 35000, - "client_rating": 4.8, - "deadline": "2026-04-30", - "required_skills": ["broadcast design", "av systems", "control room setup"] - } - ] -} -``` - ---- - -## Adding Real API Credentials (Production) - -When you're ready to search for **real opportunities**, add API keys for each platform. - -### Upwork API - -**Get Upwork API Key:** - -1. Go to https://www.upwork.com/ab/account-security/api -2. Click **Create a New Application** -3. Fill in app details: - - **Company**: Your company name - - **Application Name**: "RFP Scraper" - - **Application URL**: https://10.0.0.25:4444 -4. Accept terms and create -5. Copy **API Key** and **API Secret** - -**Add to .env:** - -```env -UPWORK_API_KEY=your_upwork_api_key_here -UPWORK_API_SECRET=your_upwork_secret_here -``` - -### Freelancer.com API - -**Get Freelancer API Key:** - -1. Go to https://www.freelancer.com/settings/integrations -2. Create new API application -3. Fill in details: - - **App Name**: "RFP Scraper" - - **Redirect URL**: https://10.0.0.25:4444/callback -4. Generate API key -5. Copy the key - -**Add to .env:** - -```env -FREELANCER_API_KEY=your_freelancer_key_here -``` - -### Toptal API - -**Get Toptal API Access:** - -1. Toptal requires direct partnership request -2. Go to https://www.toptal.com/api -3. Request API access with use case -4. Wait for approval (typically 5-7 business days) -5. Receive API key once approved - -**Add to .env:** - -```env -TOPTAL_API_KEY=your_toptal_key_here -``` - ---- - -## Switching from Mock to Real Data - -### Step 1: Add API Keys to .env - -```env -# RFP Scraper MCP Server -UPWORK_API_KEY=your_actual_key -FREELANCER_API_KEY=your_actual_key -TOPTAL_API_KEY=your_actual_key -``` - -### Step 2: Update RFP Scraper Service Code - -The RFP Scraper service is located at: -``` -/path/to/mcp-gateway/rfpscraper-mcp/src/services/ -``` - -The mock implementations need to be replaced with real API calls: - -1. **upwork.service.ts** - Replace mock data with Upwork API calls -2. **freelancer.service.ts** - Replace mock data with Freelancer API calls -3. **toptal.service.ts** - Replace mock data with Toptal API calls - -Each service file has a clearly marked section for real API integration. - -### Step 3: Rebuild and Restart - -```bash -cd /path/to/mcp-gateway -docker-compose down rfpscraper-mcp -docker-compose build rfpscraper-mcp -docker-compose up -d rfpscraper-mcp -``` - -### Step 4: Verify Real Data - -```bash -# Check logs for API calls -docker-compose logs rfpscraper-mcp -f - -# Test with MCP gateway -curl -X POST http://10.0.0.25:4444/mcp \ - -H "Authorization: Bearer YOUR_TOKEN" \ - -d '{ - "jsonrpc": "2.0", - "id": 1, - "method": "rfp_search_projects", - "params": {"keywords": "control room", "budget_min": 5000} - }' -``` - ---- - -## Development Workflow - -### Current Setup (Mock Data) - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ RFP Scraper MCP (Mock Data) β”‚ -β”‚ βœ… Returns sample projects β”‚ -β”‚ βœ… Full tool functionality β”‚ -β”‚ βœ… Testing & development ready β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ - β”‚ - β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β” - β”‚ Mock Data β”‚ - β”‚ Services β”‚ - β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - -### Production Setup (Real APIs) - -``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ RFP Scraper MCP (Real APIs) β”‚ -β”‚ βœ… Returns live opportunities β”‚ -β”‚ βœ… Full tool functionality β”‚ -β”‚ βœ… Production ready β”‚ -β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”˜ - β”‚ β”‚ -β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β” -β”‚ Upwork β”‚ β”‚Freelancerβ”‚ β”‚ Toptal β”‚ -β”‚ API β”‚ β”‚ API β”‚ β”‚ API β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ -``` - ---- - -## Rate Limits & Best Practices - -### Upwork -- **Rate Limit**: 2 requests/second -- **Caching**: Recommended (projects don't change every second) - -### Freelancer.com -- **Rate Limit**: 50 requests/minute -- **Caching**: Recommended (24 hour cache for budget ranges) - -### Toptal -- **Rate Limit**: 100 requests/minute -- **Caching**: Recommended (48 hour cache for skill matches) - -### Implementation Recommendations - -1. **Add Caching** - Store results for 1-24 hours -2. **Rate Limiting** - Implement exponential backoff -3. **Error Handling** - Gracefully degrade if one API is down -4. **Logging** - Track API calls and response times - ---- - -## Troubleshooting - -### Service shows "unhealthy" - -**With mock data:** -- Check container is running: `docker-compose ps rfpscraper-mcp` -- Check logs: `docker-compose logs rfpscraper-mcp` - -**With real APIs:** -- Verify API keys are set: `docker-compose config | grep -E "UPWORK_|FREELANCER_|TOPTAL_"` -- Check API endpoints are accessible from container -- Verify API key format (some require Bearer prefix, others don't) - -### No results returned - -**With mock data:** -- Check keywords match mock data (try "broadcast studio", "control room") -- Verify budget range overlaps with mock data ($5k-$50k range) - -**With real APIs:** -- Check API credentials are valid -- Try broader search terms -- Check API usage limits haven't been exceeded -- Verify network connectivity to API endpoints - -### API Integration Issues - -1. Check API documentation for latest endpoint format -2. Verify authentication headers (Bearer vs API-Key vs Basic) -3. Check response format matches expected schema -4. Test APIs manually first: - ```bash - curl -H "Authorization: Bearer YOUR_KEY" \ - https://api.upwork.com/v1/contractor/profile - ``` - ---- - -## Next Steps - -### Immediate (Now) -- βœ… Deploy RFP Scraper with mock data -- βœ… Test with dashboard at http://10.0.0.25:4444/dashboard -- βœ… Grant users access in user management dashboard - -### Short-term (This Week) -- Get Upwork API key -- Get Freelancer API key -- Request Toptal API access - -### Medium-term (Once APIs Obtained) -- Update service files with real API calls -- Add caching layer -- Test with real data -- Monitor API usage and costs - -### Long-term (Production) -- Add rate limiting and backoff strategies -- Implement comprehensive logging -- Set up alerts for API failures -- Optimize search filters based on actual opportunities - ---- - -## Cost Estimation - -| Platform | Tier | Cost | Rate Limit | -|----------|------|------|-----------| -| **Upwork** | Free | $0 | 2 req/sec | -| **Freelancer** | Free | $0 | 50 req/min | -| **Toptal** | Partnership | Varies | 100 req/min | - -For your use case (periodic RFP searches), **all platforms offer free tier access** that's more than sufficient. - ---- - -## Support & Documentation - -- **Upwork API Docs**: https://developers.upwork.com/ -- **Freelancer API Docs**: https://developers.freelancer.com/ -- **Toptal Partnership**: https://www.toptal.com/partnerships - -Your RFP Scraper is production-ready! Start with mock data for testing, then add real APIs when you're ready.