mcp-servers/RFP_SCRAPER_SETUP.md

RFP Scraper MCP - Setup & Configuration Guide

Overview

The RFP Scraper MCP is a custom-built tool that searches for Request For Proposal (RFP) opportunities across multiple freelance platforms. It's designed to find control room design, broadcast studio, and AV systems projects.

Current Status

✅ Ready to Deploy - Currently uses mock/sample data 🔄 Production Ready Architecture - Prepared for real API integration ⚠️ API Credentials Optional - Add when you want live scraping

---

What It Does

The RFP Scraper searches for freelance projects on:

1. Upwork - Large freelance marketplace
2. Freelancer.com - Global freelance platform
3. Toptal - High-end freelance talent network

Available Tools

rfp_search_projects
├── Search by keywords (e.g., "control room", "broadcast studio")
├── Filter by budget range
├── Sort by date, relevance, budget
└── Get paginated results

rfp_get_project_details
├── Fetch full project information
├── Client details and ratings
├── Deadline and budget
└── Required skills and experience

rfp_filter_by_budget
├── Filter projects by price range
├── Show budget breakdown
└── Cost analysis

---

Deployment

Step 1: No Changes Needed for Mock Mode

The RFP Scraper is already configured to return sample data. You can deploy and test it immediately:

cd /path/to/mcp-gateway
docker-compose up -d rfpscraper-mcp

Step 2: Verify It's Running

curl http://10.0.0.25:4444/dashboard/status | jq '.services[] | select(.name=="RFP Scraper")'

Should show:

{
  "name": "RFP Scraper",
  "status": "healthy",
  "toolCount": 3,
  "responseTime": 45
}

---

Using Mock Data (Development/Testing)

The RFP Scraper returns realistic sample data for testing:

# Example request
curl -X POST http://10.0.0.25:4444/mcp \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "rfp_search_projects",
    "params": {
      "keywords": "broadcast studio",
      "budget_min": 5000,
      "budget_max": 50000
    }
  }'

Sample response:

{
  "projects": [
    {
      "id": "upwork_12345",
      "platform": "upwork",
      "title": "Professional Broadcast Studio Setup for Sports Network",
      "budget": 35000,
      "client_rating": 4.8,
      "deadline": "2026-04-30",
      "required_skills": ["broadcast design", "av systems", "control room setup"]
    }
  ]
}

---

Adding Real API Credentials (Production)

When you're ready to search for real opportunities, add API keys for each platform.

Upwork API

Get Upwork API Key:

1. Go to https://www.upwork.com/ab/account-security/api
2. Click Create a New Application
3. Fill in app details:
   • Company: Your company name
   • Application Name: "RFP Scraper"
   • Application URL: https://10.0.0.25:4444
4. Accept terms and create
5. Copy API Key and API Secret

Add to .env:

UPWORK_API_KEY=your_upwork_api_key_here
UPWORK_API_SECRET=your_upwork_secret_here

Freelancer.com API

Get Freelancer API Key:

1. Go to https://www.freelancer.com/settings/integrations
2. Create new API application
3. Fill in details:
   • App Name: "RFP Scraper"
   • Redirect URL: https://10.0.0.25:4444/callback
4. Generate API key
5. Copy the key

Add to .env:

FREELANCER_API_KEY=your_freelancer_key_here

Toptal API

Get Toptal API Access:

1. Toptal requires direct partnership request
2. Go to https://www.toptal.com/api
3. Request API access with use case
4. Wait for approval (typically 5-7 business days)
5. Receive API key once approved

Add to .env:

TOPTAL_API_KEY=your_toptal_key_here

---

Switching from Mock to Real Data

Step 1: Add API Keys to .env

# RFP Scraper MCP Server
UPWORK_API_KEY=your_actual_key
FREELANCER_API_KEY=your_actual_key
TOPTAL_API_KEY=your_actual_key

Step 2: Update RFP Scraper Service Code

The RFP Scraper service is located at:

/path/to/mcp-gateway/rfpscraper-mcp/src/services/

The mock implementations need to be replaced with real API calls:

1. upwork.service.ts - Replace mock data with Upwork API calls
2. freelancer.service.ts - Replace mock data with Freelancer API calls
3. toptal.service.ts - Replace mock data with Toptal API calls

Each service file has a clearly marked section for real API integration.

Step 3: Rebuild and Restart

cd /path/to/mcp-gateway
docker-compose down rfpscraper-mcp
docker-compose build rfpscraper-mcp
docker-compose up -d rfpscraper-mcp

Step 4: Verify Real Data

# Check logs for API calls
docker-compose logs rfpscraper-mcp -f

# Test with MCP gateway
curl -X POST http://10.0.0.25:4444/mcp \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "rfp_search_projects",
    "params": {"keywords": "control room", "budget_min": 5000}
  }'

---

Development Workflow

Current Setup (Mock Data)

┌─────────────────────────────────────┐
│   RFP Scraper MCP (Mock Data)       │
│  ✅ Returns sample projects         │
│  ✅ Full tool functionality         │
│  ✅ Testing & development ready     │
└──────────────┬──────────────────────┘
               │
        ┌──────▼──────┐
        │ Mock Data   │
        │ Services    │
        └─────────────┘

Production Setup (Real APIs)

┌─────────────────────────────────────┐
│   RFP Scraper MCP (Real APIs)       │
│  ✅ Returns live opportunities      │
│  ✅ Full tool functionality         │
│  ✅ Production ready                │
└────┬────────────────────────────┬───┘
     │                            │
┌────▼────┐  ┌──────────┐  ┌────▼───┐
│ Upwork  │  │Freelancer│  │ Toptal │
│ API     │  │ API      │  │ API    │
└─────────┘  └──────────┘  └────────┘

---

Rate Limits & Best Practices

Upwork

• Rate Limit: 2 requests/second
• Caching: Recommended (projects don't change every second)

Freelancer.com

• Rate Limit: 50 requests/minute
• Caching: Recommended (24 hour cache for budget ranges)

Toptal

• Rate Limit: 100 requests/minute
• Caching: Recommended (48 hour cache for skill matches)

Implementation Recommendations

1. Add Caching - Store results for 1-24 hours
2. Rate Limiting - Implement exponential backoff
3. Error Handling - Gracefully degrade if one API is down
4. Logging - Track API calls and response times

---

Troubleshooting

Service shows "unhealthy"

With mock data:

• Check container is running: docker-compose ps rfpscraper-mcp
• Check logs: docker-compose logs rfpscraper-mcp

With real APIs:

• Verify API keys are set: docker-compose config | grep -E "UPWORK_|FREELANCER_|TOPTAL_"
• Check API endpoints are accessible from container
• Verify API key format (some require Bearer prefix, others don't)

No results returned

With mock data:

• Check keywords match mock data (try "broadcast studio", "control room")
• Verify budget range overlaps with mock data ($5k-$50k range)

With real APIs:

• Check API credentials are valid
• Try broader search terms
• Check API usage limits haven't been exceeded
• Verify network connectivity to API endpoints

API Integration Issues

1. Check API documentation for latest endpoint format
2. Verify authentication headers (Bearer vs API-Key vs Basic)
3. Check response format matches expected schema
4. Test APIs manually first:

   curl -H "Authorization: Bearer YOUR_KEY" \
        https://api.upwork.com/v1/contractor/profile
   

---

Next Steps

Immediate (Now)

• ✅ Deploy RFP Scraper with mock data
• ✅ Test with dashboard at http://10.0.0.25:4444/dashboard
• ✅ Grant users access in user management dashboard

Short-term (This Week)

• Get Upwork API key
• Get Freelancer API key
• Request Toptal API access

Medium-term (Once APIs Obtained)

• Update service files with real API calls
• Add caching layer
• Test with real data
• Monitor API usage and costs

Long-term (Production)

• Add rate limiting and backoff strategies
• Implement comprehensive logging
• Set up alerts for API failures
• Optimize search filters based on actual opportunities

---

Cost Estimation

Platform	Tier	Cost	Rate Limit
Upwork	Free	$0	2 req/sec
Freelancer	Free	$0	50 req/min
Toptal	Partnership	Varies	100 req/min

For your use case (periodic RFP searches), all platforms offer free tier access that's more than sufficient.

---

Support & Documentation

• Upwork API Docs: https://developers.upwork.com/
• Freelancer API Docs: https://developers.freelancer.com/
• Toptal Partnership: https://www.toptal.com/partnerships

Your RFP Scraper is production-ready! Start with mock data for testing, then add real APIs when you're ready.