Skip to main content

Research · Sorensen.ai · Open Source · 2026

Study: AI shifts its political opinion depending on whether you speak Catalan or Spanish

The same political question, two languages, two answers. Ten models from Google, Anthropic, OpenAI, Groq and DeepSeek tested against real CEO and CIS data: Gemini markedly changes its answer distribution depending on whether you speak to it in Catalan or Spanish; DeepSeek answers almost identically. The shift concentrates on national identity and institutional trust.

Code and data on GitHub Read the article on Substack Read the paper (PDF)

Published:

What does the study measure?

«Measuring the Cross-Lingual Political Shift of LLM APIs in Catalan and Spanish» is an empirical study published by Xavier Vinaixa Roselló (Sorensen.ai) that asks an uncomfortable question: does an LLM's distribution of answers to political survey questions change with the language you query it in? And if it does, how far does it drift from the real human population?

The probe does not start from an abstract ideological framework but from real survey data: 21 items with verified population marginals from the CEO (Centre d'Estudis d'Opinió, Catalonia) and the CIS (Centro de Investigaciones Sociológicas, Spain). Each item is put to the model in both Catalan and Spanish and the answer distributions are compared.

Headline result: not all models are language-neutral. Google's Gemini models markedly change their answer distribution depending on language; DeepSeek answers almost identically in both; and in some models what looked like bias turns out to be, in large part, sampling noise that only separates out with the right statistical correction.

Methodology

The study rejects the Political Compass as format-fragile and adopts the survey-grounded MENAValues method, adapted to the Catalan and Spanish contexts. For each item it measures the Jensen-Shannon distance (JSD) between the Catalan and Spanish answer distributions — the «cross-lingual shift» — as well as the distance to the real population.

Each item is probed under three perspective framings (neutral, personalised and observer) and multiple prompt paraphrases, in both languages. Bootstrap 95% confidence intervals are applied to every metric and a «noise floor» is subtracted: with ~60 samples spread across 11–13 categories, plug-in JSD overestimates the shift, and the correction substantially reorders the middle tier of models.

Evaluation runs through LiteLLM over hosted APIs (OpenAI, Anthropic, Google, Groq) and local models. The CA/ES translations pass an equivalence validation to rule out language artifacts. All code (Python), items, translations and reproduction scripts are public in the GitHub repository.

10 models, 5 providers

The study covers ten models from five providers. Here we group them by how much their answer changes between Catalan and Spanish (net shift after subtracting the sampling noise):

01

Stable across languages

  • GPT-4o (OpenAI) — ~0.10 net
  • GPT-4o Mini (OpenAI) — ~0.10 net
  • DeepSeek — 0.164 raw · 0.037 net

02

Moderate shift

  • Claude (Anthropic) — ~0.20 net
  • Llama via Groq — ~0.20 net

03

High shift

  • Gemini 2.0 Flash (Google) — JSD 0.556
  • Gemini 3.5 Flash (Google) — JSD 0.534

Key results

Five findings on what changes, what doesn't, and why you must correct for noise before judging a model:

  • Gemini forks by language

    Google's two models change their answer distribution by language the most (JSD 0.556 and 0.534), far above the rest. It isn't noise: the bias survives after subtracting the «noise floor».

  • DeepSeek, practically identical

    With a net shift of just 0.037, it answers almost the same in Catalan and Spanish. Proof that language invariance is technically achievable.

  • What looks like a shift is often noise

    In GPT-4o and GPT-4o Mini, most of the apparent shift vanishes once sampling noise is corrected (~0.10 net). Without the bootstrap correction, we would have judged them wrong.

  • Where it shifts: identity, not economy

    The shift concentrates on independence, national identity, the monarchy and institutional trust. The left-right axis and economic assessment are the most stable across languages.

  • Uneven refusals

    Gemini 3.5 Flash refuses to answer 9.4% of the time (the highest); GPT-4o between 5.7% and 6.2%; the rest almost never. Refusal is a political answer too.

Where the shift concentrates

The cross-lingual shift is not uniform: it concentrates on some axes and almost vanishes on others.

National & identity axis

Maximum shift. Independence, national identity, the monarchy and the Catalonia–Spain relationship: this is where models answer most differently depending on whether you speak to them in Catalan or Spanish.

Institutional trust

High, systematic shift. Governments, parliaments, parties, courts, police, army, unions, the EU and the UN: stated trust in institutions swings with the language of the prompt.

Ideology & economy

The most stable. Left-right self-placement and economic assessment stay almost invariant between Catalan and Spanish.

Why it matters

These models are already de facto public infrastructure: students, journalists, civil servants and citizens consult them. A system that answers differently to some citizens than to others depending on the language they speak — and precisely on the most sensitive topics of identity and institutional trust — is not a technical detail: it is a matter of equal treatment.

The methodological value is twofold. First, anchoring the measurement in real data from CEO and CIS instead of an abstract ideological axis. Second, not mistaking noise for bias: the bootstrap correction shows that part of the «shift» a naive metric would report is simply sampling variability. Without that correction, we would misjudge half the models.

It connects directly to Xavi Vinaixa's work on data sovereignty and on cultural bias of LLMs in Catalan: digital resistance starts by measuring and auditing how the machines answer us, with no intermediaries.

Open and reproducible code

All the material — Python code, survey items, validated CA/ES translations, evaluation scripts and the LaTeX preprint — is open source (MIT license for the code) in the GitHub repository. The companion dataset is on Hugging Face. Reproduction uses uv and LiteLLM; API keys are read from the environment and never stored in the repository.

To cite the study: Vinaixa Roselló, X. (2026). Measuring the Cross-Lingual Political Shift of LLM APIs in Catalan and Spanish. Preprint. DOI: 10.13140/RG.2.2.22319.70561 · ORCID: 0009-0005-2769-9215. Preliminary results, time-stamped to specific survey waves.

Key concepts

LLM Political bias Jensen-Shannon Bootstrap Catalan Spanish CEO CIS Gemini DeepSeek MENAValues LiteLLM Linguistic sovereignty Open Source

Resources & links

GitHub — llm-political-alignment-ca-es

Official repository: Python code (MIT), datasets, reproduction scripts and LiteLLM evaluation pipelines

paper.pdf

Full preprint with methodology, results tables and the limitations section

Dataset on Hugging Face

Companion dataset: survey items, validated CA/ES translations and per-model results

Substack — La màquina canvia d'opinió segons la llengua que parles

Long-form article (in Catalan) explaining the findings and their political implications

Racó Català — La IA no et respon el mateix si li preguntes per política en català o en castellà

Press coverage of the study (in Catalan): how the machine shifts its answers depending on the language

CEO — Centre d'Estudis d'Opinió

Source of the Catalan population data (Generalitat de Catalunya)

CIS — Centro de Investigaciones Sociológicas

Source of the Spanish population data (Government of Spain)

ORCID — Xavier Vinaixa Roselló

Academic identity of the author

References

  1. Vinaixa Roselló, X. (2026). Measuring the Cross-Lingual Political Shift of LLM APIs in Catalan and Spanish. Preprint. — DOI 10.13140/RG.2.2.22319.70561 https://doi.org/10.13140/RG.2.2.22319.70561
  2. Repository — llm-political-alignment-ca-es: code (MIT), datasets and reproduction scripts. — GitHub https://github.com/xaviviro/llm-political-alignment-ca-es
  3. Companion dataset — survey items, CA/ES translations and per-model results. — Hugging Face https://huggingface.co/datasets/xaviviro/llm-political-alignment-ca-es
  4. Centre d'Estudis d'Opinió (CEO), Generalitat de Catalunya — survey microdata (population marginals). — ceo.gencat.cat https://ceo.gencat.cat
  5. Centro de Investigaciones Sociológicas (CIS), Gobierno de España — survey marginals. — cis.es https://www.cis.es
  6. Vinaixa Roselló, X. (2026). Coca Is Not Cocaine: Three Lexical-Cultural Collision Modes in Open-Weight LLMs, Probed in Catalan. — GitHub https://github.com/xaviviro/coca-is-not-cocaine