Remove mcp-gateway/RFP_SCRAPER_SETUP.md
This commit is contained in:
parent
edcd2efd05
commit
73d0c5d1a1
1 changed files with 0 additions and 370 deletions
|
|
@ -1,370 +0,0 @@
|
|||
# RFP Scraper MCP - Setup & Configuration Guide
|
||||
|
||||
## Overview
|
||||
|
||||
The **RFP Scraper MCP** is a custom-built tool that searches for Request For Proposal (RFP) opportunities across multiple freelance platforms. It's designed to find control room design, broadcast studio, and AV systems projects.
|
||||
|
||||
### Current Status
|
||||
|
||||
✅ **Ready to Deploy** - Currently uses mock/sample data
|
||||
🔄 **Production Ready Architecture** - Prepared for real API integration
|
||||
⚠️ **API Credentials Optional** - Add when you want live scraping
|
||||
|
||||
---
|
||||
|
||||
## What It Does
|
||||
|
||||
The RFP Scraper searches for freelance projects on:
|
||||
|
||||
1. **Upwork** - Large freelance marketplace
|
||||
2. **Freelancer.com** - Global freelance platform
|
||||
3. **Toptal** - High-end freelance talent network
|
||||
|
||||
### Available Tools
|
||||
|
||||
```
|
||||
rfp_search_projects
|
||||
├── Search by keywords (e.g., "control room", "broadcast studio")
|
||||
├── Filter by budget range
|
||||
├── Sort by date, relevance, budget
|
||||
└── Get paginated results
|
||||
|
||||
rfp_get_project_details
|
||||
├── Fetch full project information
|
||||
├── Client details and ratings
|
||||
├── Deadline and budget
|
||||
└── Required skills and experience
|
||||
|
||||
rfp_filter_by_budget
|
||||
├── Filter projects by price range
|
||||
├── Show budget breakdown
|
||||
└── Cost analysis
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Step 1: No Changes Needed for Mock Mode
|
||||
|
||||
The RFP Scraper is **already configured** to return sample data. You can deploy and test it immediately:
|
||||
|
||||
```bash
|
||||
cd /path/to/mcp-gateway
|
||||
docker-compose up -d rfpscraper-mcp
|
||||
```
|
||||
|
||||
### Step 2: Verify It's Running
|
||||
|
||||
```bash
|
||||
curl http://10.0.0.25:4444/dashboard/status | jq '.services[] | select(.name=="RFP Scraper")'
|
||||
```
|
||||
|
||||
Should show:
|
||||
```json
|
||||
{
|
||||
"name": "RFP Scraper",
|
||||
"status": "healthy",
|
||||
"toolCount": 3,
|
||||
"responseTime": 45
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Using Mock Data (Development/Testing)
|
||||
|
||||
The RFP Scraper returns **realistic sample data** for testing:
|
||||
|
||||
```bash
|
||||
# Example request
|
||||
curl -X POST http://10.0.0.25:4444/mcp \
|
||||
-H "Authorization: Bearer YOUR_API_KEY" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "rfp_search_projects",
|
||||
"params": {
|
||||
"keywords": "broadcast studio",
|
||||
"budget_min": 5000,
|
||||
"budget_max": 50000
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
Sample response:
|
||||
```json
|
||||
{
|
||||
"projects": [
|
||||
{
|
||||
"id": "upwork_12345",
|
||||
"platform": "upwork",
|
||||
"title": "Professional Broadcast Studio Setup for Sports Network",
|
||||
"budget": 35000,
|
||||
"client_rating": 4.8,
|
||||
"deadline": "2026-04-30",
|
||||
"required_skills": ["broadcast design", "av systems", "control room setup"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding Real API Credentials (Production)
|
||||
|
||||
When you're ready to search for **real opportunities**, add API keys for each platform.
|
||||
|
||||
### Upwork API
|
||||
|
||||
**Get Upwork API Key:**
|
||||
|
||||
1. Go to https://www.upwork.com/ab/account-security/api
|
||||
2. Click **Create a New Application**
|
||||
3. Fill in app details:
|
||||
- **Company**: Your company name
|
||||
- **Application Name**: "RFP Scraper"
|
||||
- **Application URL**: https://10.0.0.25:4444
|
||||
4. Accept terms and create
|
||||
5. Copy **API Key** and **API Secret**
|
||||
|
||||
**Add to .env:**
|
||||
|
||||
```env
|
||||
UPWORK_API_KEY=your_upwork_api_key_here
|
||||
UPWORK_API_SECRET=your_upwork_secret_here
|
||||
```
|
||||
|
||||
### Freelancer.com API
|
||||
|
||||
**Get Freelancer API Key:**
|
||||
|
||||
1. Go to https://www.freelancer.com/settings/integrations
|
||||
2. Create new API application
|
||||
3. Fill in details:
|
||||
- **App Name**: "RFP Scraper"
|
||||
- **Redirect URL**: https://10.0.0.25:4444/callback
|
||||
4. Generate API key
|
||||
5. Copy the key
|
||||
|
||||
**Add to .env:**
|
||||
|
||||
```env
|
||||
FREELANCER_API_KEY=your_freelancer_key_here
|
||||
```
|
||||
|
||||
### Toptal API
|
||||
|
||||
**Get Toptal API Access:**
|
||||
|
||||
1. Toptal requires direct partnership request
|
||||
2. Go to https://www.toptal.com/api
|
||||
3. Request API access with use case
|
||||
4. Wait for approval (typically 5-7 business days)
|
||||
5. Receive API key once approved
|
||||
|
||||
**Add to .env:**
|
||||
|
||||
```env
|
||||
TOPTAL_API_KEY=your_toptal_key_here
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Switching from Mock to Real Data
|
||||
|
||||
### Step 1: Add API Keys to .env
|
||||
|
||||
```env
|
||||
# RFP Scraper MCP Server
|
||||
UPWORK_API_KEY=your_actual_key
|
||||
FREELANCER_API_KEY=your_actual_key
|
||||
TOPTAL_API_KEY=your_actual_key
|
||||
```
|
||||
|
||||
### Step 2: Update RFP Scraper Service Code
|
||||
|
||||
The RFP Scraper service is located at:
|
||||
```
|
||||
/path/to/mcp-gateway/rfpscraper-mcp/src/services/
|
||||
```
|
||||
|
||||
The mock implementations need to be replaced with real API calls:
|
||||
|
||||
1. **upwork.service.ts** - Replace mock data with Upwork API calls
|
||||
2. **freelancer.service.ts** - Replace mock data with Freelancer API calls
|
||||
3. **toptal.service.ts** - Replace mock data with Toptal API calls
|
||||
|
||||
Each service file has a clearly marked section for real API integration.
|
||||
|
||||
### Step 3: Rebuild and Restart
|
||||
|
||||
```bash
|
||||
cd /path/to/mcp-gateway
|
||||
docker-compose down rfpscraper-mcp
|
||||
docker-compose build rfpscraper-mcp
|
||||
docker-compose up -d rfpscraper-mcp
|
||||
```
|
||||
|
||||
### Step 4: Verify Real Data
|
||||
|
||||
```bash
|
||||
# Check logs for API calls
|
||||
docker-compose logs rfpscraper-mcp -f
|
||||
|
||||
# Test with MCP gateway
|
||||
curl -X POST http://10.0.0.25:4444/mcp \
|
||||
-H "Authorization: Bearer YOUR_TOKEN" \
|
||||
-d '{
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "rfp_search_projects",
|
||||
"params": {"keywords": "control room", "budget_min": 5000}
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Current Setup (Mock Data)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ RFP Scraper MCP (Mock Data) │
|
||||
│ ✅ Returns sample projects │
|
||||
│ ✅ Full tool functionality │
|
||||
│ ✅ Testing & development ready │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
┌──────▼──────┐
|
||||
│ Mock Data │
|
||||
│ Services │
|
||||
└─────────────┘
|
||||
```
|
||||
|
||||
### Production Setup (Real APIs)
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ RFP Scraper MCP (Real APIs) │
|
||||
│ ✅ Returns live opportunities │
|
||||
│ ✅ Full tool functionality │
|
||||
│ ✅ Production ready │
|
||||
└────┬────────────────────────────┬───┘
|
||||
│ │
|
||||
┌────▼────┐ ┌──────────┐ ┌────▼───┐
|
||||
│ Upwork │ │Freelancer│ │ Toptal │
|
||||
│ API │ │ API │ │ API │
|
||||
└─────────┘ └──────────┘ └────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rate Limits & Best Practices
|
||||
|
||||
### Upwork
|
||||
- **Rate Limit**: 2 requests/second
|
||||
- **Caching**: Recommended (projects don't change every second)
|
||||
|
||||
### Freelancer.com
|
||||
- **Rate Limit**: 50 requests/minute
|
||||
- **Caching**: Recommended (24 hour cache for budget ranges)
|
||||
|
||||
### Toptal
|
||||
- **Rate Limit**: 100 requests/minute
|
||||
- **Caching**: Recommended (48 hour cache for skill matches)
|
||||
|
||||
### Implementation Recommendations
|
||||
|
||||
1. **Add Caching** - Store results for 1-24 hours
|
||||
2. **Rate Limiting** - Implement exponential backoff
|
||||
3. **Error Handling** - Gracefully degrade if one API is down
|
||||
4. **Logging** - Track API calls and response times
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service shows "unhealthy"
|
||||
|
||||
**With mock data:**
|
||||
- Check container is running: `docker-compose ps rfpscraper-mcp`
|
||||
- Check logs: `docker-compose logs rfpscraper-mcp`
|
||||
|
||||
**With real APIs:**
|
||||
- Verify API keys are set: `docker-compose config | grep -E "UPWORK_|FREELANCER_|TOPTAL_"`
|
||||
- Check API endpoints are accessible from container
|
||||
- Verify API key format (some require Bearer prefix, others don't)
|
||||
|
||||
### No results returned
|
||||
|
||||
**With mock data:**
|
||||
- Check keywords match mock data (try "broadcast studio", "control room")
|
||||
- Verify budget range overlaps with mock data ($5k-$50k range)
|
||||
|
||||
**With real APIs:**
|
||||
- Check API credentials are valid
|
||||
- Try broader search terms
|
||||
- Check API usage limits haven't been exceeded
|
||||
- Verify network connectivity to API endpoints
|
||||
|
||||
### API Integration Issues
|
||||
|
||||
1. Check API documentation for latest endpoint format
|
||||
2. Verify authentication headers (Bearer vs API-Key vs Basic)
|
||||
3. Check response format matches expected schema
|
||||
4. Test APIs manually first:
|
||||
```bash
|
||||
curl -H "Authorization: Bearer YOUR_KEY" \
|
||||
https://api.upwork.com/v1/contractor/profile
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate (Now)
|
||||
- ✅ Deploy RFP Scraper with mock data
|
||||
- ✅ Test with dashboard at http://10.0.0.25:4444/dashboard
|
||||
- ✅ Grant users access in user management dashboard
|
||||
|
||||
### Short-term (This Week)
|
||||
- Get Upwork API key
|
||||
- Get Freelancer API key
|
||||
- Request Toptal API access
|
||||
|
||||
### Medium-term (Once APIs Obtained)
|
||||
- Update service files with real API calls
|
||||
- Add caching layer
|
||||
- Test with real data
|
||||
- Monitor API usage and costs
|
||||
|
||||
### Long-term (Production)
|
||||
- Add rate limiting and backoff strategies
|
||||
- Implement comprehensive logging
|
||||
- Set up alerts for API failures
|
||||
- Optimize search filters based on actual opportunities
|
||||
|
||||
---
|
||||
|
||||
## Cost Estimation
|
||||
|
||||
| Platform | Tier | Cost | Rate Limit |
|
||||
|----------|------|------|-----------|
|
||||
| **Upwork** | Free | $0 | 2 req/sec |
|
||||
| **Freelancer** | Free | $0 | 50 req/min |
|
||||
| **Toptal** | Partnership | Varies | 100 req/min |
|
||||
|
||||
For your use case (periodic RFP searches), **all platforms offer free tier access** that's more than sufficient.
|
||||
|
||||
---
|
||||
|
||||
## Support & Documentation
|
||||
|
||||
- **Upwork API Docs**: https://developers.upwork.com/
|
||||
- **Freelancer API Docs**: https://developers.freelancer.com/
|
||||
- **Toptal Partnership**: https://www.toptal.com/partnerships
|
||||
|
||||
Your RFP Scraper is production-ready! Start with mock data for testing, then add real APIs when you're ready.
|
||||
Loading…
Reference in a new issue