LLM & Models API

Overview

The LLM & Models API provides a unified interface for managing multiple LLM providers, model routing, inference, and self-hosted Ollama instances. Supported providers: openai, anthropic, google, xai, ollama, replicate.

---

LLM Backends

Backends represent configured connections to LLM providers.

Endpoints

---

List Backends

GET /v1/llm/backends

Query Parameters

Response

jsonCopy
{
  "data": [
    {
      "id": "backend_abc123",
      "name": "OpenAI Production",
      "provider": "openai",
      "status": "active",
      "base_url": "https://api.openai.com/v1",
      "model_count": 12,
      "is_default": true,
      "health": {
        "status": "healthy",
        "latency_ms": 120,
        "last_checked_at": "2025-12-07T15:00:00Z"
      },
      "created_at": "2025-11-01T10:00:00Z"
    },
    {
      "id": "backend_def456",
      "name": "Local Ollama",
      "provider": "ollama",
      "status": "active",
      "base_url": "http://10.8.0.49:11434",
      "model_count": 5,
      "is_default": false,
      "health": {
        "status": "healthy",
        "latency_ms": 15,
        "last_checked_at": "2025-12-07T15:00:00Z"
      },
      "created_at": "2025-11-15T09:00:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "per_page": 20,
    "total": 4,
    "total_pages": 1
  }
}

Example

bashCopy
curl "https://api.arcanflows.io/v1/llm/backends?provider=openai" \
  -H "Authorization: Bearer your_api_key"

---

Create Backend

POST /v1/llm/backends

Request Body

jsonCopy
{
  "name": "OpenAI Production",
  "provider": "openai",
  "base_url": "https://api.openai.com/v1",
  "credential_id": "cred_abc123",
  "is_default": true,
  "config": {
    "organization_id": "org-xxxx",
    "max_retries": 3,
    "timeout_ms": 30000
  }
}

Parameters

Response

jsonCopy
{
  "data": {
    "id": "backend_abc123",
    "name": "OpenAI Production",
    "provider": "openai",
    "status": "active",
    "base_url": "https://api.openai.com/v1",
    "model_count": 0,
    "is_default": true,
    "created_at": "2025-12-07T10:00:00Z"
  }
}

Example

bashCopy
curl -X POST "https://api.arcanflows.io/v1/llm/backends" \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Anthropic Claude",
    "provider": "anthropic",
    "credential_id": "cred_xyz789"
  }'

---

Health Check

POST /v1/llm/backends/:id/health-check

Performs a connectivity and authentication check against the provider.

Response

jsonCopy
{
  "data": {
    "backend_id": "backend_abc123",
    "status": "healthy",
    "latency_ms": 145,
    "models_available": 12,
    "message": "Connection successful",
    "checked_at": "2025-12-07T16:00:00Z"
  }
}

---

Sync Models

POST /v1/llm/backends/:id/sync-models

Discovers and syncs available models from the provider. Creates model records for newly found models and updates existing ones.

Response

jsonCopy
{
  "data": {
    "backend_id": "backend_abc123",
    "models_discovered": 15,
    "models_added": 3,
    "models_updated": 10,
    "models_removed": 2,
    "synced_at": "2025-12-07T16:05:00Z"
  }
}

---

Pull Model (Ollama)

POST /v1/llm/backends/:id/models/pull

Pulls a model to an Ollama backend. Only available for ollama provider backends.

Request Body

jsonCopy
{
  "model": "llama3.1:8b",
  "stream": false
}

Response

jsonCopy
{
  "data": {
    "backend_id": "backend_def456",
    "model": "llama3.1:8b",
    "status": "pulling",
    "message": "Model pull initiated"
  }
}

---

Delete Backend Model

DELETE /v1/llm/backends/:id/models/:model

Removes a model from the backend. For Ollama backends, this also deletes the model from the instance.

Response

HTTP/1.1 204 No Content

---

Models

Unified model registry across all backends.

Endpoints

---

List Models (Detailed)

GET /v1/llm/models

Query Parameters

Response

jsonCopy
{
  "data": [
    {
      "id": "model_abc123",
      "name": "gpt-4o",
      "display_name": "GPT-4o",
      "provider": "openai",
      "backend_id": "backend_abc123",
      "backend_name": "OpenAI Production",
      "capabilities": ["chat", "vision", "function_calling", "json_mode"],
      "context_window": 128000,
      "max_output_tokens": 16384,
      "pricing": {
        "input_per_1m_tokens": 2.50,
        "output_per_1m_tokens": 10.00,
        "currency": "usd"
      },
      "is_active": true,
      "created_at": "2025-12-01T10:00:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "per_page": 50,
    "total": 35,
    "total_pages": 1
  }
}

---

List Available Models (Simplified)

GET /v1/models

A simplified endpoint intended for agent configuration dropdowns and model selection UIs.

Response

jsonCopy
{
  "data": [
    {
      "id": "model_abc123",
      "name": "gpt-4o",
      "display_name": "GPT-4o",
      "provider": "openai",
      "capabilities": ["chat", "vision", "function_calling"],
      "context_window": 128000
    },
    {
      "id": "model_def456",
      "name": "claude-3-5-sonnet-20241022",
      "display_name": "Claude 3.5 Sonnet",
      "provider": "anthropic",
      "capabilities": ["chat", "vision", "function_calling"],
      "context_window": 200000
    },
    {
      "id": "model_ghi789",
      "name": "llama3.1:8b",
      "display_name": "Llama 3.1 8B",
      "provider": "ollama",
      "capabilities": ["chat"],
      "context_window": 131072
    }
  ]
}

---

List Providers

GET /v1/models/providers

Response

jsonCopy
{
  "data": [
    {
      "provider": "openai",
      "display_name": "OpenAI",
      "model_count": 12,
      "status": "active"
    },
    {
      "provider": "anthropic",
      "display_name": "Anthropic",
      "model_count": 6,
      "status": "active"
    },
    {
      "provider": "google",
      "display_name": "Google AI",
      "model_count": 4,
      "status": "active"
    },
    {
      "provider": "xai",
      "display_name": "xAI",
      "model_count": 3,
      "status": "active"
    },
    {
      "provider": "ollama",
      "display_name": "Ollama (Self-hosted)",
      "model_count": 5,
      "status": "active"
    },
    {
      "provider": "replicate",
      "display_name": "Replicate",
      "model_count": 8,
      "status": "active"
    }
  ]
}

---

Model Capabilities

GET /v1/models/capabilities/list

Response

jsonCopy
{
  "data": [
    { "capability": "chat", "description": "Text chat completions", "model_count": 35 },
    { "capability": "vision", "description": "Image understanding", "model_count": 12 },
    { "capability": "function_calling", "description": "Tool/function calling", "model_count": 18 },
    { "capability": "json_mode", "description": "Structured JSON output", "model_count": 15 },
    { "capability": "streaming", "description": "Streaming responses", "model_count": 30 },
    { "capability": "embeddings", "description": "Text embeddings", "model_count": 8 },
    { "capability": "image_generation", "description": "Image generation", "model_count": 5 },
    { "capability": "code", "description": "Code generation and analysis", "model_count": 20 },
    { "capability": "reasoning", "description": "Extended reasoning/thinking", "model_count": 6 }
  ]
}

---

Compare Models

POST /v1/models/compare

Request Body

jsonCopy
{
  "model_ids": ["model_abc123", "model_def456", "model_ghi789"]
}

Response

jsonCopy
{
  "data": [
    {
      "id": "model_abc123",
      "name": "gpt-4o",
      "provider": "openai",
      "context_window": 128000,
      "max_output_tokens": 16384,
      "capabilities": ["chat", "vision", "function_calling", "json_mode"],
      "pricing": { "input_per_1m_tokens": 2.50, "output_per_1m_tokens": 10.00 },
      "avg_latency_ms": 850,
      "avg_tokens_per_second": 65
    },
    {
      "id": "model_def456",
      "name": "claude-3-5-sonnet-20241022",
      "provider": "anthropic",
      "context_window": 200000,
      "max_output_tokens": 8192,
      "capabilities": ["chat", "vision", "function_calling", "json_mode"],
      "pricing": { "input_per_1m_tokens": 3.00, "output_per_1m_tokens": 15.00 },
      "avg_latency_ms": 900,
      "avg_tokens_per_second": 55
    },
    {
      "id": "model_ghi789",
      "name": "llama3.1:8b",
      "provider": "ollama",
      "context_window": 131072,
      "max_output_tokens": 4096,
      "capabilities": ["chat"],
      "pricing": { "input_per_1m_tokens": 0, "output_per_1m_tokens": 0 },
      "avg_latency_ms": 200,
      "avg_tokens_per_second": 40
    }
  ]
}

---

Routing

Intelligent model routing distributes requests across backends based on rules and equivalences.

Endpoints

---

List Routing Rules

GET /v1/llm/routing/rules

Response

jsonCopy
{
  "data": {
    "strategy": "priority",
    "fallback_enabled": true,
    "rules": [
      {
        "id": "rule_001",
        "name": "Production Traffic",
        "priority": 1,
        "condition": "backend.status == 'active' && backend.health == 'healthy'",
        "action": "route_to_default",
        "description": "Route to default backend when healthy"
      },
      {
        "id": "rule_002",
        "name": "Fallback to Ollama",
        "priority": 2,
        "condition": "default_backend.health != 'healthy'",
        "action": "route_to_backend",
        "target_backend": "backend_def456",
        "description": "Fall back to local Ollama when cloud backends are down"
      }
    ]
  }
}

---

Model Equivalences

Equivalences define which models can substitute for each other during failover.

GET /v1/llm/routing/equivalences

Response

jsonCopy
{
  "data": [
    {
      "id": "eq_001",
      "name": "GPT-4 Class",
      "models": [
        { "model": "gpt-4o", "provider": "openai", "priority": 1 },
        { "model": "claude-3-5-sonnet-20241022", "provider": "anthropic", "priority": 2 },
        { "model": "gemini-1.5-pro", "provider": "google", "priority": 3 }
      ]
    },
    {
      "id": "eq_002",
      "name": "Fast Models",
      "models": [
        { "model": "gpt-4o-mini", "provider": "openai", "priority": 1 },
        { "model": "claude-3-5-haiku-20241022", "provider": "anthropic", "priority": 2 },
        { "model": "llama3.1:8b", "provider": "ollama", "priority": 3 }
      ]
    }
  ]
}

---

Create Equivalence

POST /v1/llm/routing/equivalences

Request Body

jsonCopy
{
  "name": "Vision Models",
  "models": [
    { "model": "gpt-4o", "provider": "openai", "priority": 1 },
    { "model": "claude-3-5-sonnet-20241022", "provider": "anthropic", "priority": 2 },
    { "model": "gemini-1.5-pro", "provider": "google", "priority": 3 }
  ]
}

---

Test Routing

POST /v1/llm/routing/test

Simulates a routing decision without making an actual inference request.

Request Body

jsonCopy
{
  "model": "gpt-4o",
  "capabilities_required": ["chat", "vision"],
  "fallback": true
}

Response

jsonCopy
{
  "data": {
    "requested_model": "gpt-4o",
    "routed_model": "gpt-4o",
    "routed_backend": "backend_abc123",
    "backend_name": "OpenAI Production",
    "route_reason": "direct_match",
    "fallback_chain": [
      { "model": "claude-3-5-sonnet-20241022", "backend": "backend_xyz789", "reason": "equivalence_group" },
      { "model": "gemini-1.5-pro", "backend": "backend_ghi456", "reason": "equivalence_group" }
    ]
  }
}

---

Inference

OpenAI-compatible inference endpoints for chat completions and text generation.

Endpoints

---

Chat Completions

POST /v1/llm/chat/completions

OpenAI-compatible chat completions endpoint. Works with any configured backend and supports automatic routing.

Request Body

jsonCopy
{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Explain quantum computing in simple terms." }
  ],
  "temperature": 0.7,
  "max_tokens": 1024,
  "stream": false
}

Parameters

Response

jsonCopy
{
  "id": "chatcmpl_abc123",
  "object": "chat.completion",
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Quantum computing uses quantum bits (qubits) instead of classical bits..."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 25,
    "completion_tokens": 150,
    "total_tokens": 175
  },
  "routing": {
    "backend": "OpenAI Production",
    "provider": "openai",
    "latency_ms": 1200
  }
}

Streaming Example

bashCopy
curl -X POST "https://api.arcanflows.io/v1/llm/chat/completions" \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Streaming response uses Server-Sent Events:

data: {"id":"chatcmpl_abc","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl_abc","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: {"id":"chatcmpl_abc","choices":[{"delta":{"content":"!"},"index":0}]}

data: [DONE]

---

Text Generation

POST /v1/llm/generate

Simple text generation endpoint for non-chat use cases.

Request Body

jsonCopy
{
  "model": "gpt-4o",
  "prompt": "Write a product description for a wireless mouse:",
  "max_tokens": 200,
  "temperature": 0.8
}

Response

jsonCopy
{
  "data": {
    "model": "gpt-4o",
    "text": "Introducing the SilentGlide Pro wireless mouse...",
    "usage": {
      "prompt_tokens": 12,
      "completion_tokens": 85,
      "total_tokens": 97
    }
  }
}

---

Metrics & Statistics

Endpoints

---

Global Metrics

GET /v1/llm/metrics

Query Parameters

Response

jsonCopy
{
  "data": {
    "period": "24h",
    "total_requests": 15420,
    "total_tokens": 8540000,
    "total_cost_usd": 42.50,
    "avg_latency_ms": 920,
    "error_rate": 0.02,
    "by_provider": [
      {
        "provider": "openai",
        "requests": 8500,
        "tokens": 5200000,
        "cost_usd": 28.00,
        "avg_latency_ms": 850
      },
      {
        "provider": "anthropic",
        "requests": 4200,
        "tokens": 2100000,
        "cost_usd": 12.50,
        "avg_latency_ms": 950
      },
      {
        "provider": "ollama",
        "requests": 2720,
        "tokens": 1240000,
        "cost_usd": 0,
        "avg_latency_ms": 200
      }
    ]
  }
}

---

Backend Metrics

GET /v1/llm/metrics/backends/:id

Response

jsonCopy
{
  "data": {
    "backend_id": "backend_abc123",
    "backend_name": "OpenAI Production",
    "period": "24h",
    "requests": 8500,
    "successful": 8430,
    "failed": 70,
    "error_rate": 0.008,
    "tokens": {
      "input": 3200000,
      "output": 2000000,
      "total": 5200000
    },
    "cost_usd": 28.00,
    "latency": {
      "avg_ms": 850,
      "p50_ms": 720,
      "p95_ms": 1800,
      "p99_ms": 3200
    },
    "top_models": [
      { "model": "gpt-4o", "requests": 5000, "tokens": 3500000 },
      { "model": "gpt-4o-mini", "requests": 3500, "tokens": 1700000 }
    ]
  }
}

---

Routing Log

GET /v1/llm/routing-log

Query Parameters

Response

jsonCopy
{
  "data": [
    {
      "id": "route_001",
      "requested_model": "gpt-4o",
      "routed_model": "gpt-4o",
      "routed_backend": "OpenAI Production",
      "was_fallback": false,
      "reason": "direct_match",
      "latency_ms": 850,
      "timestamp": "2025-12-07T15:30:00Z"
    },
    {
      "id": "route_002",
      "requested_model": "gpt-4o",
      "routed_model": "claude-3-5-sonnet-20241022",
      "routed_backend": "Anthropic Production",
      "was_fallback": true,
      "reason": "primary_backend_unhealthy",
      "latency_ms": 920,
      "timestamp": "2025-12-07T15:31:00Z"
    }
  ],
  "pagination": {
    "page": 1,
    "per_page": 50,
    "total": 320,
    "total_pages": 7
  }
}

---

Ollama Instances

Dedicated management for self-hosted Ollama instances.

Endpoints

---

List Instances

GET /v1/ollama/instances

Response

jsonCopy
{
  "data": [
    {
      "id": "ollama_abc123",
      "name": "GPU Server",
      "base_url": "http://10.8.0.49:11434",
      "status": "active",
      "gpu_info": "NVIDIA RTX 4090 24GB",
      "model_count": 5,
      "health": {
        "status": "healthy",
        "latency_ms": 12,
        "last_checked_at": "2025-12-07T15:00:00Z"
      },
      "created_at": "2025-11-15T09:00:00Z"
    }
  ]
}

---

Add Instance

POST /v1/ollama/instances

Request Body

jsonCopy
{
  "name": "GPU Server",
  "base_url": "http://10.8.0.49:11434",
  "auto_discover": true,
  "config": {
    "max_concurrent": 4,
    "gpu_layers": -1
  }
}

Parameters

Example

bashCopy
curl -X POST "https://api.arcanflows.io/v1/ollama/instances" \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPU Server",
    "base_url": "http://10.8.0.49:11434",
    "auto_discover": true
  }'

---

Discover Models

POST /v1/ollama/instances/:id/discover

Queries the Ollama instance for all available models and syncs them to the platform.

Response

jsonCopy
{
  "data": {
    "instance_id": "ollama_abc123",
    "models_found": [
      {
        "name": "llama3.1:8b",
        "size_bytes": 4700000000,
        "parameter_size": "8B",
        "quantization": "Q4_0",
        "family": "llama"
      },
      {
        "name": "mistral:7b",
        "size_bytes": 4100000000,
        "parameter_size": "7B",
        "quantization": "Q4_0",
        "family": "mistral"
      },
      {
        "name": "nomic-embed-text",
        "size_bytes": 274000000,
        "parameter_size": "137M",
        "quantization": "F16",
        "family": "nomic-bert"
      }
    ],
    "models_added": 1,
    "models_updated": 2,
    "discovered_at": "2025-12-07T16:00:00Z"
  }
}

---

Instance Models

GET /v1/ollama/instances/:id/models

Response

jsonCopy
{
  "data": [
    {
      "id": "omodel_abc123",
      "name": "llama3.1:8b",
      "display_name": "Llama 3.1 8B",
      "parameter_size": "8B",
      "quantization": "Q4_0",
      "size_bytes": 4700000000,
      "family": "llama",
      "capabilities": ["chat", "code"],
      "context_window": 131072,
      "is_active": true,
      "last_used_at": "2025-12-07T14:30:00Z"
    }
  ]
}

---

All Ollama Models

GET /v1/ollama/models

Lists all Ollama models across all instances.

Query Parameters

---

Model Capabilities

GET /v1/ollama/models/capabilities/:name

Returns capabilities for a specific Ollama model by name.

Response

jsonCopy
{
  "data": {
    "model": "llama3.1:8b",
    "capabilities": ["chat", "code", "function_calling"],
    "context_window": 131072,
    "supports_streaming": true,
    "supports_json_mode": true,
    "supports_vision": false,
    "supports_tool_calling": true,
    "parameter_size": "8B"
  }
}

---

Code Examples

Python SDK

pythonCopy
from arcanflows import Arcanflows

client = Arcanflows(api_key='your_api_key')

# Chat completion
response = client.llm.chat_completions(
    model='gpt-4o',
    messages=[
        {'role': 'system', 'content': 'You are a helpful assistant.'},
        {'role': 'user', 'content': 'What is quantum computing?'}
    ],
    temperature=0.7
)

print(response.choices[0].message.content)

# List backends
backends = client.llm.list_backends()
for b in backends.data:
    print(f"{b.name} ({b.provider}): {b.status}")

# Health check a backend
health = client.llm.health_check('backend_abc123')
print(f"Status: {health.status}, Latency: {health.latency_ms}ms")

# Compare models
comparison = client.models.compare(['model_abc123', 'model_def456'])
for m in comparison.data:
    print(f"{m.name}: {m.context_window} ctx, {m.avg_latency_ms}ms")

# Add Ollama instance
instance = client.ollama.add_instance(
    name='Local GPU',
    base_url='http://10.8.0.49:11434',
    auto_discover=True
)

JavaScript SDK

javascriptCopy
import { ArcanFlows } from '@arcanflows/sdk';

const client = new Arcanflows({ apiKey: 'your_api_key' });

// Chat completion
const response = await client.llm.chatCompletions({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'What is quantum computing?' },
  ],
  temperature: 0.7,
});

console.log(response.choices[0].message.content);

// Streaming chat completion
const stream = await client.llm.chatCompletions({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Tell me a story' }],
  stream: true,
});

for await (const chunk of stream) {
  if (chunk.choices[0]?.delta?.content) {
    process.stdout.write(chunk.choices[0].delta.content);
  }
}

// List providers and models
const providers = await client.models.listProviders();
console.log('Providers:', providers.data.map(p => p.provider));

// Test routing
const route = await client.llm.testRouting({
  model: 'gpt-4o',
  capabilitiesRequired: ['chat', 'vision'],
  fallback: true,
});

console.log('Routed to:', route.routedModel, 'on', route.routedBackend);

// Manage Ollama
const instances = await client.ollama.listInstances();
await client.ollama.discoverModels(instances.data[0].id);

cURL Examples

bashCopy
# Chat completion
curl -X POST "https://api.arcanflows.io/v1/llm/chat/completions" \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

# Health check a backend
curl -X POST "https://api.arcanflows.io/v1/llm/backends/backend_abc123/health-check" \
  -H "Authorization: Bearer your_api_key"

# Sync models from provider
curl -X POST "https://api.arcanflows.io/v1/llm/backends/backend_abc123/sync-models" \
  -H "Authorization: Bearer your_api_key"

# Pull an Ollama model
curl -X POST "https://api.arcanflows.io/v1/llm/backends/backend_def456/models/pull" \
  -H "Authorization: Bearer your_api_key" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3.1:8b"}'

# Get LLM metrics
curl "https://api.arcanflows.io/v1/llm/metrics?period=24h&group_by=provider" \
  -H "Authorization: Bearer your_api_key"