mcp-servers/mcp-gateway/RFP_SCRAPER_SETUP.md

# RFP Scraper MCP - Setup & Configuration Guide

## Overview

The **RFP Scraper MCP** is a custom-built tool that searches for Request For Proposal (RFP) opportunities across multiple freelance platforms. It's designed to find control room design, broadcast studio, and AV systems projects.

### Current Status

✅ **Ready to Deploy** - Currently uses mock/sample data
🔄 **Production Ready Architecture** - Prepared for real API integration
⚠️ **API Credentials Optional** - Add when you want live scraping

---

## What It Does

The RFP Scraper searches for freelance projects on:

1. **Upwork** - Large freelance marketplace
2. **Freelancer.com** - Global freelance platform
3. **Toptal** - High-end freelance talent network

### Available Tools

```
rfp_search_projects
├── Search by keywords (e.g., "control room", "broadcast studio")
├── Filter by budget range
├── Sort by date, relevance, budget
└── Get paginated results

rfp_get_project_details
├── Fetch full project information
├── Client details and ratings
├── Deadline and budget
└── Required skills and experience

rfp_filter_by_budget
├── Filter projects by price range
├── Show budget breakdown
└── Cost analysis
```

---

## Deployment

### Step 1: No Changes Needed for Mock Mode

The RFP Scraper is **already configured** to return sample data. You can deploy and test it immediately:

```bash
cd /path/to/mcp-gateway
docker-compose up -d rfpscraper-mcp
```

### Step 2: Verify It's Running

```bash
curl http://10.0.0.25:4444/dashboard/status | jq '.services[] | select(.name=="RFP Scraper")'
```

Should show:
```json
{
  "name": "RFP Scraper",
  "status": "healthy",
  "toolCount": 3,
  "responseTime": 45
}
```

---

## Using Mock Data (Development/Testing)

The RFP Scraper returns **realistic sample data** for testing:

```bash
# Example request
curl -X POST http://10.0.0.25:4444/mcp \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "rfp_search_projects",
    "params": {
      "keywords": "broadcast studio",
      "budget_min": 5000,
      "budget_max": 50000
    }
  }'
```

Sample response:
```json
{
  "projects": [
    {
      "id": "upwork_12345",
      "platform": "upwork",
      "title": "Professional Broadcast Studio Setup for Sports Network",
      "budget": 35000,
      "client_rating": 4.8,
      "deadline": "2026-04-30",
      "required_skills": ["broadcast design", "av systems", "control room setup"]
    }
  ]
}
```

---

## Adding Real API Credentials (Production)

When you're ready to search for **real opportunities**, add API keys for each platform.

### Upwork API

**Get Upwork API Key:**

1. Go to https://www.upwork.com/ab/account-security/api
2. Click **Create a New Application**
3. Fill in app details:
   - **Company**: Your company name
   - **Application Name**: "RFP Scraper"
   - **Application URL**: https://10.0.0.25:4444
4. Accept terms and create
5. Copy **API Key** and **API Secret**

**Add to .env:**

```env
UPWORK_API_KEY=your_upwork_api_key_here
UPWORK_API_SECRET=your_upwork_secret_here
```

### Freelancer.com API

**Get Freelancer API Key:**

1. Go to https://www.freelancer.com/settings/integrations
2. Create new API application
3. Fill in details:
   - **App Name**: "RFP Scraper"
   - **Redirect URL**: https://10.0.0.25:4444/callback
4. Generate API key
5. Copy the key

**Add to .env:**

```env
FREELANCER_API_KEY=your_freelancer_key_here
```

### Toptal API

**Get Toptal API Access:**

1. Toptal requires direct partnership request
2. Go to https://www.toptal.com/api
3. Request API access with use case
4. Wait for approval (typically 5-7 business days)
5. Receive API key once approved

**Add to .env:**

```env
TOPTAL_API_KEY=your_toptal_key_here
```

---

## Switching from Mock to Real Data

### Step 1: Add API Keys to .env

```env
# RFP Scraper MCP Server
UPWORK_API_KEY=your_actual_key
FREELANCER_API_KEY=your_actual_key
TOPTAL_API_KEY=your_actual_key
```

### Step 2: Update RFP Scraper Service Code

The RFP Scraper service is located at:
```
/path/to/mcp-gateway/rfpscraper-mcp/src/services/
```

The mock implementations need to be replaced with real API calls:

1. **upwork.service.ts** - Replace mock data with Upwork API calls
2. **freelancer.service.ts** - Replace mock data with Freelancer API calls
3. **toptal.service.ts** - Replace mock data with Toptal API calls

Each service file has a clearly marked section for real API integration.

### Step 3: Rebuild and Restart

```bash
cd /path/to/mcp-gateway
docker-compose down rfpscraper-mcp
docker-compose build rfpscraper-mcp
docker-compose up -d rfpscraper-mcp
```

### Step 4: Verify Real Data

```bash
# Check logs for API calls
docker-compose logs rfpscraper-mcp -f

# Test with MCP gateway
curl -X POST http://10.0.0.25:4444/mcp \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "rfp_search_projects",
    "params": {"keywords": "control room", "budget_min": 5000}
  }'
```

---

## Development Workflow

### Current Setup (Mock Data)

```
┌─────────────────────────────────────┐
│   RFP Scraper MCP (Mock Data)       │
│  ✅ Returns sample projects         │
│  ✅ Full tool functionality         │
│  ✅ Testing & development ready     │
└──────────────┬──────────────────────┘
               │
        ┌──────▼──────┐
        │ Mock Data   │
        │ Services    │
        └─────────────┘
```

### Production Setup (Real APIs)

```
┌─────────────────────────────────────┐
│   RFP Scraper MCP (Real APIs)       │
│  ✅ Returns live opportunities      │
│  ✅ Full tool functionality         │
│  ✅ Production ready                │
└────┬────────────────────────────┬───┘
     │                            │
┌────▼────┐  ┌──────────┐  ┌────▼───┐
│ Upwork  │  │Freelancer│  │ Toptal │
│ API     │  │ API      │  │ API    │
└─────────┘  └──────────┘  └────────┘
```

---

## Rate Limits & Best Practices

### Upwork
- **Rate Limit**: 2 requests/second
- **Caching**: Recommended (projects don't change every second)

### Freelancer.com
- **Rate Limit**: 50 requests/minute
- **Caching**: Recommended (24 hour cache for budget ranges)

### Toptal
- **Rate Limit**: 100 requests/minute
- **Caching**: Recommended (48 hour cache for skill matches)

### Implementation Recommendations

1. **Add Caching** - Store results for 1-24 hours
2. **Rate Limiting** - Implement exponential backoff
3. **Error Handling** - Gracefully degrade if one API is down
4. **Logging** - Track API calls and response times

---

## Troubleshooting

### Service shows "unhealthy"

**With mock data:**
- Check container is running: `docker-compose ps rfpscraper-mcp`
- Check logs: `docker-compose logs rfpscraper-mcp`

**With real APIs:**
- Verify API keys are set: `docker-compose config | grep -E "UPWORK_|FREELANCER_|TOPTAL_"`
- Check API endpoints are accessible from container
- Verify API key format (some require Bearer prefix, others don't)

### No results returned

**With mock data:**
- Check keywords match mock data (try "broadcast studio", "control room")
- Verify budget range overlaps with mock data ($5k-$50k range)

**With real APIs:**
- Check API credentials are valid
- Try broader search terms
- Check API usage limits haven't been exceeded
- Verify network connectivity to API endpoints

### API Integration Issues

1. Check API documentation for latest endpoint format
2. Verify authentication headers (Bearer vs API-Key vs Basic)
3. Check response format matches expected schema
4. Test APIs manually first:
   ```bash
   curl -H "Authorization: Bearer YOUR_KEY" \
        https://api.upwork.com/v1/contractor/profile
   ```

---

## Next Steps

### Immediate (Now)
- ✅ Deploy RFP Scraper with mock data
- ✅ Test with dashboard at http://10.0.0.25:4444/dashboard
- ✅ Grant users access in user management dashboard

### Short-term (This Week)
- Get Upwork API key
- Get Freelancer API key
- Request Toptal API access

### Medium-term (Once APIs Obtained)
- Update service files with real API calls
- Add caching layer
- Test with real data
- Monitor API usage and costs

### Long-term (Production)
- Add rate limiting and backoff strategies
- Implement comprehensive logging
- Set up alerts for API failures
- Optimize search filters based on actual opportunities

---

## Cost Estimation

| Platform | Tier | Cost | Rate Limit |
|----------|------|------|-----------|
| **Upwork** | Free | $0 | 2 req/sec |
| **Freelancer** | Free | $0 | 50 req/min |
| **Toptal** | Partnership | Varies | 100 req/min |

For your use case (periodic RFP searches), **all platforms offer free tier access** that's more than sufficient.

---

## Support & Documentation

- **Upwork API Docs**: https://developers.upwork.com/
- **Freelancer API Docs**: https://developers.freelancer.com/
- **Toptal Partnership**: https://www.toptal.com/partnerships

Your RFP Scraper is production-ready! Start with mock data for testing, then add real APIs when you're ready.