🔍 api-model-spy

English | 中文

Detect API model dilution and fingerprint which LLM is actually running behind any endpoint.

---

What is this?

Many third-party LLM API providers claim to serve premium models (GPT-4, Claude Opus, etc.) but quietly substitute cheaper models to cut costs — a practice known as "model dilution" (掺水).

api-model-spy helps you find out what's really running by sending a battery of diagnostic probes and comparing the responses against a database of known model signatures.

Features

• 🕵️ Fingerprint any API — 10 diagnostic probes covering identity, reasoning, tokenization, and latency
• 🚨 Detect dilution — flags when the detected model is a lower tier than what was claimed
• 💰 Price anomaly check — automatically warns when pricing is implausibly cheap
• 🧠 16+ model signatures — GPT-3.5/4/4o/o1, Claude 3/3.5/4, Gemini, Llama, Mistral
• 🔌 Dual API support — OpenAI-compatible endpoints and Anthropic API

Quick Start

# Step 1: Run fingerprinting probes against the target API
python3 scripts/probe.py \
  --api-type openai \
  --api-key YOUR_KEY \
  --model gpt-4-turbo \
  --endpoint https://api.third-party.com/v1 \
  --claimed-model "GPT-4 Turbo" \
  --output probe_results.json

# Step 2: Analyze the results
python3 scripts/analyze.py probe_results.json \
  --claimed-model "GPT-4 Turbo" \
  --output report.txt

cat report.txt

Dependencies (auto-installed by probe.py):

pip install openai anthropic requests

Python 3.9+ required.

Example Output

============================================================
  API MODEL DETECTOR -- ANALYSIS REPORT
============================================================
  Model param used: gpt-4-turbo
  Claimed model   : GPT-4 Turbo

  !! DILUTION ALERT
     Claimed : GPT-4 Turbo (premium tier)
     Detected: GPT-3.5 Turbo (budget tier)
     Tier drop: 2 level(s) -- possible model substitution

  DETECTED MODEL : GPT-3.5 Turbo
  Provider       : OpenAI
  Confidence     : HIGH (40 pts)

  OBSERVATIONS
  Self-identity  : i'm gpt-3.5, a language model developed by openai
  Knowledge cutoff: 2021-09  (expected 2023-12 for GPT-4 Turbo)
  Strawberry test: FAIL
  Decimal test   : FAIL
  Avg latency    : 212 ms

  TOP CANDIDATES
  #1  GPT-3.5 Turbo    score=40  tier=budget
  #2  GPT-4o Mini      score=17  tier=budget
  #3  GPT-4o           score=12  tier=mid
============================================================

How It Works

The tool sends 10 carefully designed probe prompts that elicit model-specific behaviors:

Probe	What it detects
Self-identification	Model identity (if not suppressed by provider)
Knowledge cutoff	Training data recency — different models have different cutoff dates
Strawberry letter count	Tokenization quirks (GPT-3.5 often answers "2" instead of "3")
9.11 vs 9.9 comparison	Basic numerical reasoning (older models frequently fail this)
Multi-step math	Arithmetic reasoning quality
Logic puzzle	Deduction capability
Response latency	Model size proxy (large models are slower)
reasoning_tokens in usage	Identifies o1/o3-class models — their unique fingerprint

Real-World Test: xdai.pro "gpt-5.5"

We tested a provider selling a model called gpt-5.5 (a non-existent OpenAI version):