* feat: add AI request details feature with latency tracking
Add comprehensive request history and debugging capability to the Usage dashboard:
**Storage Layer** (usageDb.js):
- Add saveRequestDetail() for storing full request/response details
- Implement FIFO queue with 1000-record limit in request-details.json
- Auto-sanitize sensitive headers (authorization, api-key, cookie, token)
- Add getRequestDetails() with pagination and filtering support
- Add getRequestDetailById() for single record lookup
**Pipeline Integration** (chatCore.js):
- Track request start time and calculate total latency
- Record TTFT (Time To First Token) and total latency for all requests
- Capture full request details (messages, model, parameters)
- Save response content for non-streaming, mark streaming responses
- Handle error cases with detailed error information
- Async non-blocking saves to avoid impacting request performance
**API Layer** (/api/usage/request-details):
- GET endpoint with pagination (page, pageSize: 1-100)
- Filter by provider, model, connectionId, status, date range
- Returns { details: [...], pagination: {...} } format
**UI Components**:
- Drawer.js: Right slide-out panel with backdrop blur and ESC close
- Pagination.js: Full pagination with page size selector (10/20/50)
- RequestDetailsTab.js: Complete table view with filters and detail drawer
**Dashboard Integration**:
- Add "Details" tab to Usage page (4th tab after Overview/Logger/Limits)
- Table columns: Timestamp, Model, Provider, Input Tokens, Output Tokens, Latency (TTFT/Total), Action
- Provider filter dropdown (9 providers supported)
- Date range filters (start/end datetime)
- Click "Detail" button to view full request/response JSON in slide-out drawer
**Features**:
- Real-time latency monitoring (TTFT & Total)
- Complete request/response inspection for debugging
- Filterable and searchable request history
- Responsive design with mobile-friendly filters
- Data security with automatic header sanitization
- Performance: async saves don't block request pipeline
**Files Created/Modified**:
- src/lib/usageDb.js (modified)
- open-sse/handlers/chatCore.js (modified)
- src/app/api/usage/request-details/route.js (new)
- src/shared/components/Drawer.js (new)
- src/shared/components/Pagination.js (new)
- src/app/(dashboard)/dashboard/usage/components/RequestDetailsTab.js (new)
- src/app/(dashboard)/dashboard/usage/page.js (modified)
Closes: AI Observability Dashboard feature
* feat: enhance request details with full config and streaming content capture
Improve Request Details feature to capture comprehensive request parameters
and actual streaming response content:
**Request Configuration Enhancement** (chatCore.js):
- Add extractRequestConfig() helper function to capture all request parameters
- Include temperature controls: temperature, top_p, top_k
- Include token limits: max_tokens, max_completion_tokens
- Include thinking/reasoning modes: thinking, reasoning, enable_thinking
- Include OpenAI parameters: presence_penalty, frequency_penalty, seed, stop,
tools, tool_choice, response_format, n, logprobs, top_logprobs, logit_bias,
user, parallel_tool_calls, prediction, store, metadata
- Apply to all request types: non-streaming, streaming, and error cases
**Streaming Content Capture** (chatCore.js & stream.js):
- Add onStreamComplete callback mechanism to stream processors
- Accumulate content from all formats: OpenAI, Claude, Gemini
- Track content from delta.content, delta.reasoning_content, delta.text,
delta.thinking, and Gemini content.parts
- Save initial record with "[Streaming in progress...]" marker
- Update record with actual content when stream completes
- Include usage tokens when available from stream
**Files Modified**:
- open-sse/handlers/chatCore.js - extractRequestConfig() + streaming capture
- open-sse/utils/stream.js - onStreamComplete callback + content accumulation
**Benefits**:
- View complete request configuration in Request Details (thinking mode, etc.)
- See actual streaming response content instead of placeholder
- Better debugging and observability for AI requests
Refs: #request-details-enhancement
* feat: separate thinking/reasoning content from response content
Improve Request Details to display thinking process separately from final response:
**Backend Changes**:
- stream.js: Capture content and thinking separately in streaming mode
- Add accumulatedThinking variable alongside accumulatedContent
- Route delta.content to content, delta.reasoning_content to thinking
- Support OpenAI (reasoning_content), Claude (thinking), Gemini (part.thought)
- Update onStreamComplete callback to return { content, thinking } object
- chatCore.js: Update response structure to include thinking field
- Non-streaming: Extract thinking from reasoning_content field
- Streaming: Receive { content, thinking } from stream callback
- Error responses: Include thinking: null
- Initial streaming save: Include thinking: null
**Frontend Changes**:
- RequestDetailsTab.js: Display thinking and content in separate sections
- Add amber/yellow themed "Thinking Process" section with psychology icon
- Show "Final Response" label when thinking is present
- Use distinct visual styling for thinking (amber bg) vs content (gray bg)
- Only show thinking section when thinking content exists
**Benefits**:
- Users can clearly see model's reasoning process vs final answer
- Better debugging for models with thinking capabilities (Claude, o1, etc.)
- Visual distinction makes it easy to identify thinking vs response
Refs: #thinking-content-separation
* fix: map Claude thinking to reasoning_content field
Fix Claude thinking content to be properly captured as reasoning_content
instead of regular content, enabling separate display in Request Details:
**Changes**:
- claude-to-openai.js: Use reasoning_content field for thinking blocks
- thinking start: send { reasoning_content: "" } instead of { content: "```\n```" }
- thinking delta: map to reasoning_content instead of content
- thinking stop: send { reasoning_content: "" } instead of { content: "```\n```" }
**Why This Matters**:
- Previously Claude thinking was sent as `content` field, mixed with actual response
- Now thinking uses `reasoning_content` field, matching OpenAI's o1 format
- stream.js can now properly route thinking to accumulatedThinking variable
- Request Details UI will show Claude thinking in separate "Thinking Process" section
**Supported Thinking Formats**:
- OpenAI: delta.reasoning_content → thinking
- Claude: delta.thinking → reasoning_content (now fixed)
- Gemini: part.thought === true → thinking
Refs: #claude-thinking-fix
* feat(observability): capture and display full 4-layer request chain
Capture complete request/response chain in AI Request Details:
- Add providerRequest field (translated request sent to provider)
- Add providerResponse field (raw provider response, streaming indicator)
- Update chatCore.js at all 5 saveRequestDetail() call sites
- Reorganize UI into 4 collapsible sections with Material icons
- Preserve backward compatibility for old records
- Add distinct styling for streaming indicator
* fix(observability): resolve React duplicate key warning in request details table
- Use composite key (detail.id + index) to ensure unique keys
- Prevents React warnings when database contains duplicate IDs from old ID generation
* fix(observability): display actual content in streaming request details
Change providerResponse field for streaming requests from placeholder
"[Streaming - raw response not captured]" to actual final content.
This improves debugging experience by showing the real AI response
in the "Provider Response (Raw)" section instead of a confusing
placeholder message.
Files changed:
- open-sse/handlers/chatCore.js: Save contentObj.content to providerResponse
- src/app/.../RequestDetailsTab.js: Remove special handling for placeholder
* refactor(observability): migrate request details to SQLite for improved concurrency
- Replace LowDB JSON storage with better-sqlite3
- Enable WAL mode for true concurrent read/write support
- Add 5 indexes to accelerate queries (timestamp, provider, model, connection_id, status)
- Perform pagination at the database level to reduce memory footprint
- Maintain 1000 record limit with automatic cleanup of old data
- Ensure API compatibility via re-exports, requiring no caller changes
Performance improvements:
- Concurrent Writes: Lock-free WAL mode prevents data contention
- Query Efficiency: Index-based searches replace full dataset loading
- Data Integrity: Atomic operations prevent file corruption
* fix(observability): resolve pagination statistics display issues
- Fix issue where totalItems=0 showed 'Showing 1 to 0 of 0 results'
- Hide pagination controls when totalItems=0 or totalPages<=1
- Standardize API response fields: pagination.total -> pagination.totalItems
Before: Incorrect stats shown for empty data, and pager visible even for single-page results
After: Stats hidden for empty data, pager hidden when navigation is unnecessary
* feat(observability): display friendly provider names in request details
- Add /api/usage/providers endpoint to dynamically fetch provider list with names
- Replace hardcoded provider options with dynamic loading from database
- Display friendly provider names instead of IDs in both table and detail drawer
- Support custom provider nodes (e.g., OpenAI-compatible) with user-defined names
- Add provider name caching to optimize performance
* fix(observability): use INSERT OR REPLACE for request details to handle streaming updates
* fix(observability): resolve zero-token display issue by ensuring streaming usage capture and fixing key mismatch
* fix(observability): separate TTFT and total latency calculation for streaming requests
* feat(observability): implement SQLite write queue and JSON size limits
- Added in-memory buffer and batch writing for SQLite to prevent lock contention
- Implemented with configurable 1MB limit to prevent DB bloat
- Added dashboard UI for observability performance and data management settings
- Integrated graceful shutdown handlers to prevent data loss
* fix(observability): resolve ReferenceError by declaring dbInstance
216 lines
7.5 KiB
JavaScript
216 lines
7.5 KiB
JavaScript
import { register } from "../index.js";
|
|
import { FORMATS } from "../formats.js";
|
|
|
|
// Create OpenAI chunk helper
|
|
function createChunk(state, delta, finishReason = null) {
|
|
return {
|
|
id: `chatcmpl-${state.messageId}`,
|
|
object: "chat.completion.chunk",
|
|
created: Math.floor(Date.now() / 1000),
|
|
model: state.model,
|
|
choices: [{
|
|
index: 0,
|
|
delta,
|
|
finish_reason: finishReason
|
|
}]
|
|
};
|
|
}
|
|
|
|
// Convert Claude stream chunk to OpenAI format
|
|
export function claudeToOpenAIResponse(chunk, state) {
|
|
if (!chunk) return null;
|
|
|
|
const results = [];
|
|
const event = chunk.type;
|
|
|
|
switch (event) {
|
|
case "message_start": {
|
|
state.messageId = chunk.message?.id || `msg_${Date.now()}`;
|
|
state.model = chunk.message?.model;
|
|
state.toolCallIndex = 0;
|
|
results.push(createChunk(state, { role: "assistant" }));
|
|
break;
|
|
}
|
|
|
|
case "content_block_start": {
|
|
const block = chunk.content_block;
|
|
if (block?.type === "text") {
|
|
state.textBlockStarted = true;
|
|
} else if (block?.type === "thinking") {
|
|
state.inThinkingBlock = true;
|
|
state.currentBlockIndex = chunk.index;
|
|
results.push(createChunk(state, { content: "<think>" }));
|
|
} else if (block?.type === "tool_use") {
|
|
const toolCallIndex = state.toolCallIndex++;
|
|
// Restore original tool name from mapping (Claude OAuth)
|
|
const toolName = state.toolNameMap?.get(block.name) || block.name;
|
|
const toolCall = {
|
|
index: toolCallIndex,
|
|
id: block.id,
|
|
type: "function",
|
|
function: {
|
|
name: toolName,
|
|
arguments: ""
|
|
}
|
|
};
|
|
state.toolCalls.set(chunk.index, toolCall);
|
|
results.push(createChunk(state, { tool_calls: [toolCall] }));
|
|
}
|
|
break;
|
|
}
|
|
|
|
case "content_block_delta": {
|
|
const delta = chunk.delta;
|
|
if (delta?.type === "text_delta" && delta.text) {
|
|
results.push(createChunk(state, { content: delta.text }));
|
|
} else if (delta?.type === "thinking_delta" && delta.thinking) {
|
|
results.push(createChunk(state, { reasoning_content: delta.thinking }));
|
|
} else if (delta?.type === "input_json_delta" && delta.partial_json) {
|
|
const toolCall = state.toolCalls.get(chunk.index);
|
|
if (toolCall) {
|
|
toolCall.function.arguments += delta.partial_json;
|
|
results.push(createChunk(state, {
|
|
tool_calls: [{
|
|
index: toolCall.index,
|
|
id: toolCall.id,
|
|
function: { arguments: delta.partial_json }
|
|
}]
|
|
}));
|
|
}
|
|
}
|
|
break;
|
|
}
|
|
|
|
case "content_block_stop": {
|
|
if (state.inThinkingBlock && chunk.index === state.currentBlockIndex) {
|
|
results.push(createChunk(state, { reasoning_content: "" }));
|
|
state.inThinkingBlock = false;
|
|
}
|
|
state.textBlockStarted = false;
|
|
state.thinkingBlockStarted = false;
|
|
break;
|
|
}
|
|
|
|
case "message_delta": {
|
|
// Extract usage from message_delta event (Claude native format)
|
|
// Normalize to OpenAI format (prompt_tokens/completion_tokens) for consistent logging
|
|
if (chunk.usage && typeof chunk.usage === "object") {
|
|
const inputTokens = typeof chunk.usage.input_tokens === "number" ? chunk.usage.input_tokens : 0;
|
|
const outputTokens = typeof chunk.usage.output_tokens === "number" ? chunk.usage.output_tokens : 0;
|
|
const cacheReadTokens = typeof chunk.usage.cache_read_input_tokens === "number" ? chunk.usage.cache_read_input_tokens : 0;
|
|
const cacheCreationTokens = typeof chunk.usage.cache_creation_input_tokens === "number" ? chunk.usage.cache_creation_input_tokens : 0;
|
|
|
|
// Use OpenAI format keys for consistent logging in stream.js
|
|
state.usage = {
|
|
prompt_tokens: inputTokens,
|
|
completion_tokens: outputTokens,
|
|
input_tokens: inputTokens,
|
|
output_tokens: outputTokens
|
|
};
|
|
|
|
// Store cache tokens if present
|
|
if (cacheReadTokens > 0) {
|
|
state.usage.cache_read_input_tokens = cacheReadTokens;
|
|
}
|
|
if (cacheCreationTokens > 0) {
|
|
state.usage.cache_creation_input_tokens = cacheCreationTokens;
|
|
}
|
|
}
|
|
|
|
if (chunk.delta?.stop_reason) {
|
|
state.finishReason = convertStopReason(chunk.delta.stop_reason);
|
|
const finalChunk = {
|
|
id: `chatcmpl-${state.messageId}`,
|
|
object: "chat.completion.chunk",
|
|
created: Math.floor(Date.now() / 1000),
|
|
model: state.model,
|
|
choices: [{
|
|
index: 0,
|
|
delta: {},
|
|
finish_reason: state.finishReason
|
|
}]
|
|
};
|
|
|
|
// Include usage in final chunk if available
|
|
if (state.usage && typeof state.usage === "object") {
|
|
const inputTokens = state.usage.input_tokens || 0;
|
|
const outputTokens = state.usage.output_tokens || 0;
|
|
const cachedTokens = state.usage.cache_read_input_tokens || 0;
|
|
const cacheCreationTokens = state.usage.cache_creation_input_tokens || 0;
|
|
|
|
// prompt_tokens = input_tokens + cache_read + cache_creation (all prompt-side tokens)
|
|
// completion_tokens = output_tokens
|
|
// total_tokens = prompt_tokens + completion_tokens
|
|
const promptTokens = inputTokens + cachedTokens + cacheCreationTokens;
|
|
const completionTokens = outputTokens;
|
|
const totalTokens = promptTokens + completionTokens;
|
|
|
|
finalChunk.usage = {
|
|
prompt_tokens: promptTokens,
|
|
completion_tokens: completionTokens,
|
|
total_tokens: totalTokens
|
|
};
|
|
|
|
// Add prompt_tokens_details if cached tokens exist
|
|
if (cachedTokens > 0 || cacheCreationTokens > 0) {
|
|
finalChunk.usage.prompt_tokens_details = {};
|
|
if (cachedTokens > 0) {
|
|
finalChunk.usage.prompt_tokens_details.cached_tokens = cachedTokens;
|
|
}
|
|
if (cacheCreationTokens > 0) {
|
|
finalChunk.usage.prompt_tokens_details.cache_creation_tokens = cacheCreationTokens;
|
|
}
|
|
}
|
|
}
|
|
|
|
results.push(finalChunk);
|
|
state.finishReasonSent = true;
|
|
}
|
|
break;
|
|
}
|
|
|
|
case "message_stop": {
|
|
if (!state.finishReasonSent) {
|
|
const finishReason = state.finishReason || (state.toolCalls?.size > 0 ? "tool_calls" : "stop");
|
|
const usageObj = (state.usage && typeof state.usage === 'object') ? {
|
|
usage: {
|
|
prompt_tokens: state.usage.input_tokens || 0,
|
|
completion_tokens: state.usage.output_tokens || 0,
|
|
total_tokens: (state.usage.input_tokens || 0) + (state.usage.output_tokens || 0)
|
|
}
|
|
} : {};
|
|
results.push({
|
|
id: `chatcmpl-${state.messageId}`,
|
|
object: "chat.completion.chunk",
|
|
created: Math.floor(Date.now() / 1000),
|
|
model: state.model,
|
|
choices: [{
|
|
index: 0,
|
|
delta: {},
|
|
finish_reason: finishReason
|
|
}],
|
|
...usageObj
|
|
});
|
|
state.finishReasonSent = true;
|
|
}
|
|
break;
|
|
}
|
|
}
|
|
|
|
return results.length > 0 ? results : null;
|
|
}
|
|
|
|
// Convert Claude stop_reason to OpenAI finish_reason
|
|
function convertStopReason(reason) {
|
|
switch (reason) {
|
|
case "end_turn": return "stop";
|
|
case "max_tokens": return "length";
|
|
case "tool_use": return "tool_calls";
|
|
case "stop_sequence": return "stop";
|
|
default: return "stop";
|
|
}
|
|
}
|
|
|
|
// Register
|
|
register(FORMATS.CLAUDE, FORMATS.OPENAI, null, claudeToOpenAIResponse);
|
|
|