A Tool for Rating Text/Image/Audio Stimuli via 'LLMs'

Evaluates stimuli using Large Language Models. Supports multiple LLM providers: 'OpenAI', 'Anthropic', 'Ollama', 'LM Studio', 'DeepSeek', 'Groq', 'Mistral', and 'OpenAI-compatible' endpoints. Stimuli: plain text, local image/audio files, or image URLs. Audio is transcribed via 'OpenAI Whisper' before rating. Supports numeric, text, and raw return types.

chatRater 1.3.1

A Tool for Rating Text/Image/Audio Stimuli via Large Language Models

What's New in 1.3.1

CRAN maintenance release — no new user-facing features. 1.3.1 fixes two issues surfaced by R CMD check on CRAN incoming:

• stats::aggregate() import added to NAMESPACE (was flagged as undefined global function in rate_openai_weighted()).
• rank argument of alignment() is now documented in man/alignment.Rd (was triggering a Codoc mismatches WARNING).
• inst/CITATION is now a single APA 7 entry pointing at the PsyArXiv preprint (doi:10.31234/osf.io/mje6w_v1).

R CMD check is clean on CRAN incoming (Windows Server 2022, R-devel r90190, ucrt): Status: OK.

The probability-weighted scoring mode (method = "weighted", top_logprobs, include_probs) introduced in 1.3.0 remains the headline feature of this release line — see the What's New in 1.3.0 section below for details.

Key Features

• Multi-modal input: Plain text, local image files, image URLs, and audio files (transcribed via Whisper)
• 20+ LLM providers: OpenAI, Anthropic, DeepSeek, Groq, Mistral, Ollama, LM Studio, OpenRouter, and any OpenAI-compatible endpoint
• Flexible output: Extract numeric ratings, full text responses, or raw API output
• Batch processing: Rate multiple stimuli in one call
• Local model support: Run entirely offline with Ollama or LM Studio (no API key needed)

Supported Providers

Installation

# Install from CRAN (production version)
install.packages("chatRater")
pak::pkg_install("chatRater")

# Install from GitHub (development version)
remotes::install_github("ShiyangZheng/chatRater")

Quick Start

Using Cloud Providers (OpenAI, Anthropic, etc.)

library(chatRater)

# Basic usage with OpenAI
stim <- 'The early bird catches the worm'
res <- generate_ratings(
  model = 'gpt-4o',
  stim = stim,
  provider = 'openai',
  api_key = Sys.getenv("OPENAI_API_KEY"),
  prompt = 'You are an expert in figurative language.',
  question = 'Rate the creativity of this phrase on a scale of 1-10:',
  scale = '1-10'
)

# Using Anthropic Claude
res <- generate_ratings(
  model = 'claude-sonnet-4-20250514',
  stim = stim,
  provider = 'anthropic',
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  scale = '1-5'
)

Using Local Models (No API Key Needed!)

# Make sure Ollama is running first
# Download from: https://ollama.com

res <- generate_ratings(
  stim = 'Bite the bullet',
  provider = 'ollama',
  model = 'llama3.2',
  scale = '1-7',
  n_iterations = 3
)

# Or with LM Studio (run on port 1234 by default)
res <- generate_ratings(
  stim = 'Hit the nail on the head',
  provider = 'lmstudio',
  model = 'your-model-name',
  scale = '1-5'
)

Batch Processing

stim_list <- c('Kick the bucket', 'Beat around the bush', 'Cut to the chase')

results <- generate_ratings_for_all(
  stim_list = stim_list,
  provider = 'ollama',
  model = 'llama3.2',
  scale = '1-7',
  n_iterations = 5
)

Image Rating

# Rate an image from URL
res <- generate_ratings(
  stim = 'https://example.com/image.jpg',
  provider = 'openai',
  model = 'gpt-4o',
  api_key = Sys.getenv("OPENAI_API_KEY"),
  question = 'Rate the visual quality:',
  scale = '1-10'
)

# Rate a local image file
res <- generate_ratings(
  stim = '/path/to/image.png',
  provider = 'anthropic',
  api_key = Sys.getenv("ANTHROPIC_API_KEY"),
  scale = '1-5'
)

Audio Rating (New in 1.3.0)

# Audio is transcribed via OpenAI Whisper, then rated
res <- generate_ratings(
  stim = '/path/to/audio.mp3',
  provider = 'openai',
  model = 'gpt-4o',
  api_key = Sys.getenv("OPENAI_API_KEY"),
  prompt = 'You are rating audio transcripts.',
  question = 'Rate the formality of this speech:',
  scale = '1-10'
)

Controlling Return Type (New in 1.3.0)

# Numeric (default): extracts numbers from LLM response
res <- generate_ratings(
  stim = 'test',
  provider = 'ollama',
  model = 'llama3.2',
  return_type = 'numeric',
  scale = '1-10'
)

# Text: returns full LLM response text
res <- generate_ratings(
  stim = 'test',
  provider = 'ollama',
  model = 'llama3.2',
  return_type = 'text'
)

# Raw: returns raw API response
res <- generate_ratings(
  stim = 'test',
  provider = 'ollama',
  model = 'llama3.2',
  return_type = 'raw'
)

Configuration

Setting API Keys

# Option 1: Set environment variables in .Renviron
# OPENAI_API_KEY=sk-...
# ANTHROPIC_API_KEY=sk-ant-...

# Option 2: Pass directly
generate_ratings(
  api_key = 'sk-...',
  ...
)

# Option 3: Use Sys.getenv()
generate_ratings(
  api_key = Sys.getenv("OPENAI_API_KEY"),
  ...
)

Custom Endpoints

# For vLLM or other OpenAI-compatible servers
res <- generate_ratings(
  stim = 'test',
  provider = 'openai_compatible',
  base_url = 'http://localhost:8080/v1',
  api_key = NULL,  # or your API key if required
  model = 'your-model'
)

Citation

If you use chatRater in your research, please cite the associated preprint (APA 7):

Zheng, S. (2026, May 16). chatRater: Validating LLM-Generated Psycholinguistic Norms with Probability-Weighted Scoring [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/mje6w_v1

You can also get these citations from within R:

citation("chatRater")

Dependencies

chatRater 1.3.1 depends on:

• llmcoder (>= 1.2.0): LLM integration backend
• base64enc: For encoding local files
• tools: For file extension detection
• httr2: For HTTP requests
• curl: For file uploads
• jsonlite: For JSON parsing