diff --git a/mcp-gateway/RFP_SCRAPER_SETUP.md b/mcp-gateway/RFP_SCRAPER_SETUP.md new file mode 100644 index 0000000..c82f9a7 --- /dev/null +++ b/mcp-gateway/RFP_SCRAPER_SETUP.md @@ -0,0 +1,370 @@ +# RFP Scraper MCP - Setup & Configuration Guide + +## Overview + +The **RFP Scraper MCP** is a custom-built tool that searches for Request For Proposal (RFP) opportunities across multiple freelance platforms. It's designed to find control room design, broadcast studio, and AV systems projects. + +### Current Status + +βœ… **Ready to Deploy** - Currently uses mock/sample data +πŸ”„ **Production Ready Architecture** - Prepared for real API integration +⚠️ **API Credentials Optional** - Add when you want live scraping + +--- + +## What It Does + +The RFP Scraper searches for freelance projects on: + +1. **Upwork** - Large freelance marketplace +2. **Freelancer.com** - Global freelance platform +3. **Toptal** - High-end freelance talent network + +### Available Tools + +``` +rfp_search_projects +β”œβ”€β”€ Search by keywords (e.g., "control room", "broadcast studio") +β”œβ”€β”€ Filter by budget range +β”œβ”€β”€ Sort by date, relevance, budget +└── Get paginated results + +rfp_get_project_details +β”œβ”€β”€ Fetch full project information +β”œβ”€β”€ Client details and ratings +β”œβ”€β”€ Deadline and budget +└── Required skills and experience + +rfp_filter_by_budget +β”œβ”€β”€ Filter projects by price range +β”œβ”€β”€ Show budget breakdown +└── Cost analysis +``` + +--- + +## Deployment + +### Step 1: No Changes Needed for Mock Mode + +The RFP Scraper is **already configured** to return sample data. You can deploy and test it immediately: + +```bash +cd /path/to/mcp-gateway +docker-compose up -d rfpscraper-mcp +``` + +### Step 2: Verify It's Running + +```bash +curl http://10.0.0.25:4444/dashboard/status | jq '.services[] | select(.name=="RFP Scraper")' +``` + +Should show: +```json +{ + "name": "RFP Scraper", + "status": "healthy", + "toolCount": 3, + "responseTime": 45 +} +``` + +--- + +## Using Mock Data (Development/Testing) + +The RFP Scraper returns **realistic sample data** for testing: + +```bash +# Example request +curl -X POST http://10.0.0.25:4444/mcp \ + -H "Authorization: Bearer YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "rfp_search_projects", + "params": { + "keywords": "broadcast studio", + "budget_min": 5000, + "budget_max": 50000 + } + }' +``` + +Sample response: +```json +{ + "projects": [ + { + "id": "upwork_12345", + "platform": "upwork", + "title": "Professional Broadcast Studio Setup for Sports Network", + "budget": 35000, + "client_rating": 4.8, + "deadline": "2026-04-30", + "required_skills": ["broadcast design", "av systems", "control room setup"] + } + ] +} +``` + +--- + +## Adding Real API Credentials (Production) + +When you're ready to search for **real opportunities**, add API keys for each platform. + +### Upwork API + +**Get Upwork API Key:** + +1. Go to https://www.upwork.com/ab/account-security/api +2. Click **Create a New Application** +3. Fill in app details: + - **Company**: Your company name + - **Application Name**: "RFP Scraper" + - **Application URL**: https://10.0.0.25:4444 +4. Accept terms and create +5. Copy **API Key** and **API Secret** + +**Add to .env:** + +```env +UPWORK_API_KEY=your_upwork_api_key_here +UPWORK_API_SECRET=your_upwork_secret_here +``` + +### Freelancer.com API + +**Get Freelancer API Key:** + +1. Go to https://www.freelancer.com/settings/integrations +2. Create new API application +3. Fill in details: + - **App Name**: "RFP Scraper" + - **Redirect URL**: https://10.0.0.25:4444/callback +4. Generate API key +5. Copy the key + +**Add to .env:** + +```env +FREELANCER_API_KEY=your_freelancer_key_here +``` + +### Toptal API + +**Get Toptal API Access:** + +1. Toptal requires direct partnership request +2. Go to https://www.toptal.com/api +3. Request API access with use case +4. Wait for approval (typically 5-7 business days) +5. Receive API key once approved + +**Add to .env:** + +```env +TOPTAL_API_KEY=your_toptal_key_here +``` + +--- + +## Switching from Mock to Real Data + +### Step 1: Add API Keys to .env + +```env +# RFP Scraper MCP Server +UPWORK_API_KEY=your_actual_key +FREELANCER_API_KEY=your_actual_key +TOPTAL_API_KEY=your_actual_key +``` + +### Step 2: Update RFP Scraper Service Code + +The RFP Scraper service is located at: +``` +/path/to/mcp-gateway/rfpscraper-mcp/src/services/ +``` + +The mock implementations need to be replaced with real API calls: + +1. **upwork.service.ts** - Replace mock data with Upwork API calls +2. **freelancer.service.ts** - Replace mock data with Freelancer API calls +3. **toptal.service.ts** - Replace mock data with Toptal API calls + +Each service file has a clearly marked section for real API integration. + +### Step 3: Rebuild and Restart + +```bash +cd /path/to/mcp-gateway +docker-compose down rfpscraper-mcp +docker-compose build rfpscraper-mcp +docker-compose up -d rfpscraper-mcp +``` + +### Step 4: Verify Real Data + +```bash +# Check logs for API calls +docker-compose logs rfpscraper-mcp -f + +# Test with MCP gateway +curl -X POST http://10.0.0.25:4444/mcp \ + -H "Authorization: Bearer YOUR_TOKEN" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "rfp_search_projects", + "params": {"keywords": "control room", "budget_min": 5000} + }' +``` + +--- + +## Development Workflow + +### Current Setup (Mock Data) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ RFP Scraper MCP (Mock Data) β”‚ +β”‚ βœ… Returns sample projects β”‚ +β”‚ βœ… Full tool functionality β”‚ +β”‚ βœ… Testing & development ready β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β” + β”‚ Mock Data β”‚ + β”‚ Services β”‚ + β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +### Production Setup (Real APIs) + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ RFP Scraper MCP (Real APIs) β”‚ +β”‚ βœ… Returns live opportunities β”‚ +β”‚ βœ… Full tool functionality β”‚ +β”‚ βœ… Production ready β”‚ +β””β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”˜ + β”‚ β”‚ +β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β–Όβ”€β”€β”€β” +β”‚ Upwork β”‚ β”‚Freelancerβ”‚ β”‚ Toptal β”‚ +β”‚ API β”‚ β”‚ API β”‚ β”‚ API β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +--- + +## Rate Limits & Best Practices + +### Upwork +- **Rate Limit**: 2 requests/second +- **Caching**: Recommended (projects don't change every second) + +### Freelancer.com +- **Rate Limit**: 50 requests/minute +- **Caching**: Recommended (24 hour cache for budget ranges) + +### Toptal +- **Rate Limit**: 100 requests/minute +- **Caching**: Recommended (48 hour cache for skill matches) + +### Implementation Recommendations + +1. **Add Caching** - Store results for 1-24 hours +2. **Rate Limiting** - Implement exponential backoff +3. **Error Handling** - Gracefully degrade if one API is down +4. **Logging** - Track API calls and response times + +--- + +## Troubleshooting + +### Service shows "unhealthy" + +**With mock data:** +- Check container is running: `docker-compose ps rfpscraper-mcp` +- Check logs: `docker-compose logs rfpscraper-mcp` + +**With real APIs:** +- Verify API keys are set: `docker-compose config | grep -E "UPWORK_|FREELANCER_|TOPTAL_"` +- Check API endpoints are accessible from container +- Verify API key format (some require Bearer prefix, others don't) + +### No results returned + +**With mock data:** +- Check keywords match mock data (try "broadcast studio", "control room") +- Verify budget range overlaps with mock data ($5k-$50k range) + +**With real APIs:** +- Check API credentials are valid +- Try broader search terms +- Check API usage limits haven't been exceeded +- Verify network connectivity to API endpoints + +### API Integration Issues + +1. Check API documentation for latest endpoint format +2. Verify authentication headers (Bearer vs API-Key vs Basic) +3. Check response format matches expected schema +4. Test APIs manually first: + ```bash + curl -H "Authorization: Bearer YOUR_KEY" \ + https://api.upwork.com/v1/contractor/profile + ``` + +--- + +## Next Steps + +### Immediate (Now) +- βœ… Deploy RFP Scraper with mock data +- βœ… Test with dashboard at http://10.0.0.25:4444/dashboard +- βœ… Grant users access in user management dashboard + +### Short-term (This Week) +- Get Upwork API key +- Get Freelancer API key +- Request Toptal API access + +### Medium-term (Once APIs Obtained) +- Update service files with real API calls +- Add caching layer +- Test with real data +- Monitor API usage and costs + +### Long-term (Production) +- Add rate limiting and backoff strategies +- Implement comprehensive logging +- Set up alerts for API failures +- Optimize search filters based on actual opportunities + +--- + +## Cost Estimation + +| Platform | Tier | Cost | Rate Limit | +|----------|------|------|-----------| +| **Upwork** | Free | $0 | 2 req/sec | +| **Freelancer** | Free | $0 | 50 req/min | +| **Toptal** | Partnership | Varies | 100 req/min | + +For your use case (periodic RFP searches), **all platforms offer free tier access** that's more than sufficient. + +--- + +## Support & Documentation + +- **Upwork API Docs**: https://developers.upwork.com/ +- **Freelancer API Docs**: https://developers.freelancer.com/ +- **Toptal Partnership**: https://www.toptal.com/partnerships + +Your RFP Scraper is production-ready! Start with mock data for testing, then add real APIs when you're ready.