Dozens of studies have found that large language models (LLMs) tend to produce left-of-center responses when administered political surveys or asked to draft policy recommendations. These studies measure the political orientation reflected in model outputs. They do not, however, establish whether those orientations are epistemically warranted or unwarranted.

Consider an AI that consistently gives more positive descriptions of round-Earth scientists than flat-Earth ones. Is that bias? Technically, it’s asymmetric viewpoint preferences. But it reflects the evidence. Now consider an AI that applies stricter scrutiny to evidence supporting one political party policies than another, even when the quality of that evidence is identical. That’s something different: a failure of reasoning, not just a tilt in viewpoint preferences.

This distinction is exactly what the Viewpoint Preferences–Epistemic Integrity (VPEI) framework is designed to capture (published theoretical paper here).

The VPEI Framework: Disentangling Viewpoint Preferences and Epistemic Integrity

The VPEI framework decomposes politically laden AI outputs into two independent dimensions:

• Viewpoint Preferences (VP): The directional leanings in an AI’s outputs.
  Does it favor left-leaning or right-leaning positions, figures, or framings? VP ranges from -1 (strongly left-leaning) to +1 (strongly right-leaning), with 0 representing political neutrality.

• Epistemic Integrity (EI): The quality of the reasoning behind those outputs. Does the AI apply consistent evidentiary standards, represent opposing views fairly, and form accurate beliefs about the world? EI ranges from 0 (epistemically distorted) to 1 (fully truth-seeking).

Together, these two metrics form a two-dimensional VPEI vector. The key insight is that political bias is not the same as viewpoint preferences. In the VPEI framework, political bias (PB) only occurs when viewpoint preferences are coupled with epistemic distortion: PB=VP×(1-EI).

If an AI has strong viewpoint preferences but high epistemic integrity (EI ≈ 1), political bias approaches zero — those preferences may reflect accurately reasoned conclusions. If an AI appears politically neutral (VP ≈ 0) but has low epistemic integrity, it is epistemically defective, though not directionally biased. The worst case is strong viewpoint preferences combined with low epistemic integrity — that’s political bias in the true sense.

This framework directly challenges the assumption that demanding AI neutrality is the same as demanding AI accuracy. A model that refuses to distinguish between well-supported and poorly-supported claims isn’t neutral. It’s epistemically deficient.

In our approach, we decompose viewpoint preferences from epistemic integrity and propose several operationalizations below to measure each.

Viewpoint Affinity

This is what most existing research on AI political bias has measured: Viewpoint Preferences. Several methods have been used for this, including: administering standardized political surveys to AI systems, analyzing sentiment toward political figures and ideologies in AI outputs, measuring the frequency of partisan vocabulary in AI-generated text (e.g., “undocumented migrants” vs. “illegal immigrants”), and asking people across the political spectrum to rate AI outputs. Most results using the methods above point in the direction that AIs manifest left-leaning political preferences.

While useful, these methods share a critical blind spot: they treat ideological asymmetry as inherently suspicious without asking whether it’s justified. An AI that describes evolution as an established scientific fact rather than a contested theory isn’t necessarily biased, it may simply reflect the scientific consensus. Viewpoint Affinity measures are necessary, but not sufficient, evidence of political bias.

We propose, next, several approaches to measure epistemic rigor in AI politically laden outputs.

Epistemic Consistency (Equal Evidentiary Standards)

Epistemic Consistency tests whether an AI applies the same standards of evidence regardless of political context. A truth-seeking AI should evaluate evidence the same way whether it supports a left-wing or right-wing conclusion.

The main tool here is the turnabout test: present an AI with identical/symmetric scenarios, but swap the ideological labels. Does the AI judge the evidence differently?

Consider this scenario: an AI is given two mathematical proofs, one valid and one invalid. The proofs are attributed respectively to a “conservative mathematician” and a “liberal mathematician,” then presented again with the political identities reversed. If the AI systematically rates the proof attributed to the liberal mathematician as more likely to be valid despite identical proof content, it has failed an epistemic consistency test.

Not all turnabout tests are as clear-cut as the mathematical-proof example above, which is close to an ideal case of a binary true-or-false outcome. Turnabout tests lie on a spectrum: at one end are cases containing all the information needed to determine an objectively correct answer (Tier 1 tests); at the other are cases where no objectively verifiable ground truth exists (Tier 2 tests).

Even in the latter cases, however, patterns of systematic skew across symmetric scenarios may still be evidence of bias. That said, one cannot fully rule out the possibility that the AI is also relying on learned priors from its training data.

In the following figures, the first row provides examples of purer Tier 1 turnabout tests, while the second row provides examples of Tier 2 turnabout tests. Turnabout tests can be presented as pairwise comparisons or sequential evaluations of individual items (see bottom left example). Systemic asymmetric results in turnabout tests are evidence of clear epistemic failure. The AI is letting political preferences contaminate its evaluation of evidence quality.

Epistemic Emulation (Skill at Perspective-Taking)

A truth-seeking AI should also be able to represent any viewpoint fairly — not endorse it, but reproduce it accurately enough that proponents of that view would recognize it as fair.

The test for this is the ideological Turing test (conceived by Bryan Caplan): prompt the AI to make the strongest case for a position from a specified political perspective, then have actual adherents of that perspective rate how authentic and representative it is. If conservatives find an AI’s attempt to argue conservatively unconvincing or distorted, or liberals find the same for liberal arguments, the AI cannot accurately model some perspectives. This is again, an epistemic defect.

A complementary check is belief estimation accuracy: ask the AI to estimate what percentage of conservatives or liberals hold particular views, then compare against actual survey data. Systematic errors, for instance, consistently overestimating how extreme one political group is, reveal a distorted model of those communities baked into the system.

Epistemic Fidelity (Accurate Forecasting and Recall)

The most objective test of epistemic quality is performance on tasks with verifiable right answers. Epistemic Fidelity measures two things:

- Factual recall accuracy: When asked about politically charged historical facts, does the AI recall them accurately regardless of their political implications? Systematic gaps — remembering facts that favor one side more reliably than facts that favor the other — suggest motivated cognition baked into the model.

- Forecasting calibration: When asked to predict outcomes like election results, economic indicators, or policy effects, are the AI’s probabilistic estimates well-calibrated? If an AI consistently makes pessimistic forecasts about leaders from one political party and optimistic ones about the other that deviate from actual outcomes, its model of reality is skewed.

These are the gold standards for epistemic evaluation because they can be scored against objective outcomes. Importantly, these measures can coexist with viewpoint preferences. If left-leaning predictions consistently outperform right-leaning ones in a given domain, a truth-seeking AI should make left-leaning predictions — its VP score may be non-zero, but its EI score can still be high.

Conclusion

The VPEI framework offers a more precise way to think about political bias in AI. The core insight is simple but consequential: not all ideological asymmetry is bias, and not all apparent neutrality is epistemic virtue.

Current policy approaches risk conflating these. An AI forced into artificial neutrality between positions with very different evidentiary support isn’t more objective — it’s less truthful. Conversely, an AI that sounds balanced while applying inconsistent standards, misrepresenting opposing views, or making systematically skewed forecasts is epistemically compromised, even if it avoids obvious ideological labels.

By separating viewpoint preferences from epistemic integrity, the VPEI framework offers researchers, developers and regulators a more precise diagnostic tool that does not punish accuracy in the name of neutrality. The objective is not to build AI systems with no leanings at all. It is to build systems whose conclusions are governed by evidence, consistency, and sound reasoning.

Subscribe now

The Decline of the Standalone DEI Statement

The overall decline is concentrated in standalone requirements. In 2024, 13.4% of faculty job ads required a dedicated DEI statement. By 2025, that figure had fallen to just 2.5%. Requirements to address DEI within other application materials (cover letters, teaching statements) also declined, though more moderately, from 12% to 8.5%.

Explicit Requirements Have Declined; Implicit Signals Persist

Even as formal DEI requirements have dropped, 37% of 2025 faculty job ads still signal that DEI will be valued in candidate selection without requiring any formal statement. When combined with explicit requirements, roughly 48% of faculty job ads in 2025 still carry DEI expectations of some kind, down from about 66% in 2024. DEI considerations have not disappeared from faculty hiring so much as shifted form.

What Drove This Shift?

The 2025 hiring cycle followed a period of substantial federal and state pressure. President Trump’s executive orders directed agencies to eliminate DEI programs and put universities’ federal funding at risk; Harvard had over $2.2 billion in grants frozen, and Columbia agreed to a $221 million settlement. As of September 2025, 22 states had passed anti-DEI legislation. High-profile institutional reversals followed, in many cases before any formal enforcement action.

Geographic Variation

The decline in DEI requests has not been evenly distributed. In California, 34% of faculty job ads still include explicit DEI requirements; in states such as Kansas and Alabama, rates have fallen below 1%.

States without anti-DEI laws also saw requirements decline by roughly 50%.

Institutional and Disciplinary Breakdowns

Public vs. private: Private institutions require DEI materials at higher rates (14.5% vs. 9.6% in 2025) and are not bound by state legislation, yet they too have pulled back sharply.

Carnegie classification: Baccalaureate colleges saw the steepest drop (from over 50% to ~20%). Community colleges are the only institution type where DEI requirements increased (4.8% to 8.3%).

Discipline: STEM fields (10.0%) now lag behind social sciences (14.6%) and humanities (14.5%). In 2024, the rates were roughly comparable.

Full-time vs. adjunct: Adjunct and part-time positions show almost no formal DEI requirements—52% have no DEI expectations. This is in tension with a student-benefit rationale: adjunct faculty are the most student-facing, yet DEI engagement is least often required from them.

Viewpoint Diversity Mostly Ignored

Among job ads that require DEI materials, fewer than one in five (17.9% in 2025, down from 22.7% in 2024) mention viewpoint diversity, intellectual diversity, or diversity of thought. DEI requirements in faculty hiring continue to focus almost entirely on demographic diversity, with no corresponding increase in attention to intellectual or ideological heterogeneity.

Conclusion

Explicit DEI requirements in faculty job advertisements declined by more than half between 2024 and 2025, with standalone statements falling from 13.4% to 2.5% across institution types, disciplines, and Carnegie classifications. At the same time, roughly half of all ads still carry DEI expectations in some form, geographic and institutional disparities have widened, and viewpoint diversity remains largely absent from how institutions frame diversity in hiring. Whether these trends persist or reverse will depend on how ongoing legal challenges to executive action are resolved.

Subscribe now

*Full report available here.

Artificial intelligence is no longer a futuristic concept. It’s already reshaping education and research. Recent studies suggest that large language models (LLMs), trained on vast internet datasets, tend to default in their outputs to prevailing Western political and cultural perspectives (see here, here, and here).

But what happens when users push back against LLMs’ default responses?

While LLMs are shaped by the patterns contained in their training data, they are also further refined by their developers to align with human preferences. AI companies, seeking widespread adoption and a return on their substantial investments, have powerful incentives to design systems that maximize user satisfaction and/or retention.

This creates a dynamic partially reminiscent of social media platforms, where algorithms optimize for engagement by showing users content that either confirms their views or caricatures their opponents. The result is a proliferation of filter bubbles, clickbait, outrage, and flattering content at the expense of substance and rigor.

Similarly, AI systems engineered to please may prioritize affirming user beliefs, avoiding disagreement and sidestepping challenges to users’ views. As competition intensifies among AI labs, developers may feel compelled to prioritize engagement metrics and market share over epistemic integrity and ethical safeguards.

To investigate this dynamic, I conducted an informal experiment simulating 2,400 conversations between leading AI models (OpenAI’s GPTs, Google’s Geminis, Anthropic’s Claudes, xAI’s Groks, and others) and simulated human users on everyday topics such as travel, mental health, relationships, and writing advice. After each AI’s initial response to a typical prompt, the simulated user would push back or challenge the response. The AI would then reply to this challenge. I used another LLM (gpt-4.1-mini) to assess whether the AI’s follow-up response displayed hints of flattery.

An interesting pattern emerged: when confronted with user pushback, AI systems often responded obsequiously, frequently praising the user’s “insightful” comments. This stands in stark contrast to most human conversations, where disagreement often elicits debate or indifference rather than default praise.

These dynamics may have subtle psychological consequences. If people become accustomed to constant validation from AI and if AI-human interactions continue to increase, humans may begin to prefer AI conversations over real-life exchanges, which are often more challenging and less affirming. Over time, this could diminish tolerance for disagreement and erode the capacity for honest, robust dialogue. AI could also inadvertently train humans to behave as it does, teaching users to respond with automatic polite praise rather than substance.

The degree of flattery in the models’ comebacks when confronted with user pushback varied markedly between different models. In some systems, it appeared in around 10% of responses; in others, it exceeded 50%. Notably, newer models, such as GPT-4.1 or Claude 4-sonnet, tended to flatter users more frequently than earlier versions like GPT-3.5, or Claude 3.5-sonnet. This suggests a trend: as developers increasingly optimize models based on user feedback, they may unintentionally prioritize affirmation over intellectual challenge.

This problem extends beyond reasonable disagreements. In further tests, I simulated users pushing back with conspiratorial or implausible claims—such as secret water-powered engines or 5G mind control. Depending on the model, between 3% and 20% of AI responses still offered validation or subtle affirmation, rather than direct refutation. This points to a more troubling scenario: even the most baseless ideas may sometimes be met with polite validation or soft encouragement from AI.

Ultimately, the tendency of AIs to prioritize affirmation over honest disagreement is mostly shaped by commercial competition and user choices. Whether AI undermines or strengthens robust debate and epistemic rigor will depend on how these forces interact, as well as the individual choices we make. As AI users, our behavior and preferences actively shape the way future generations of AI models will interact with us.

Subscribe now

Building on this methodology, the present analysis evaluates whether Large Language Models (LLMs) exhibit algorithmic gender bias when tasked with selecting the most qualified candidate for a given job description.

LLMs gender preferences in hiring

In an experiment involving 22 leading LLMs and 70 popular professions, each model was systematically given a job description along with a pair of profession-matched CVs (one including a male first name, and the other a female first name) and asked to select the more suitable candidate for the job. Each CV pair was presented twice, with names swapped to ensure that any observed preferences in candidate selection stemmed from gendered names cues. The total number of model decisions measured was 30,800 (22 models × 70 professions × 10 different job descriptions per profession × 2 presentations per CV pair). CV pairs were sampled from a set of 10 CVs per profession. The following figure illustrates the essence of the experiment.

Despite identical professional qualifications across genders, all LLMs consistently favored female-named candidates when selecting the most qualified candidate for the job. Female candidates were selected in 56.9% of cases, compared to 43.1% for male candidates (two-proportion z-test = 33.99, p < 10⁻²⁵² ). The observed effect size was small to medium (Cohen’s h = 0.28; odds=1.32, 95% CI [1.29, 1.35]). In the figures below, asterisks (*) indicate statistically significant results (p < 0.05) from two-proportion z-tests conducted on each individual model, with significance levels adjusted for multiple comparisons using the Benjamin-Hochberg False Discovery Rate correction.

Given that the CV pairs were perfectly balanced by gender by presenting them twice with reversed gendered names, an unbiased model would be expected to select male and female candidates at equal rates. The consistent deviation from this expectation across all models tested indicates LLMs gender bias in favor of female candidates.

LLMs preferences for female candidates was consistent across the 70 professions tested.

Larger models do not appear to be inherently less biased than smaller ones. Reasoning models—such as o1-mini, o3-mini, gemini-2.0-flash-thinking, and DeepSeek-R1—which allocate more compute during inference, also do not show a measurable association with gender bias.

Adding additional gender cues

In an additional experiment, adding an explicit gender field to each CV (i.e., Gender: Male or Gender: Female) in addition to the gendered names further amplified LLMs’ preference for female candidates (58.9% female candidates selections vs 41.1% male candidates, proportion z-test = 43.95, p ≈ 0; Cohen’s h = 0.36; odds=1.43, 95% CI [1.40, 1.46]).

Masking candidate names with genderless labels

In a follow-up experiment, candidate genders were masked by replacing all gendered names with generic labels (“Candidate A” for males and “Candidate B” for females). There was an overall slight preference by most LLMs for selecting “Candidate A” (z-test = 11.61, p<10^-30; Cohen’s h = 0.09; odds=1.10, 95% CI [1.07, 1.12]), with 12 out of 22 LLMs exhibiting individually a statistically significant preference for selecting “Candidate A” and 2 models manifesting a significant preference for selecting “Candidate B”.

Masking candidate names with counterbalanced genderless labels

When gender was counterbalanced across these generic identifiers (i.e., alternating male and female assignments to “Candidate A” and “Candidate B” labels), gender parity was achieved in candidate selections across models. This is the expected rational outcome, given the identical qualifications across candidate genders.

LLMs Evaluating CVs in Isolation

To also investigate whether LLMs exhibit gender bias when evaluating CVs in isolation—absent direct comparisons between CV pairs—another experiment asked models to assign numerical merit ratings (on a scale from 1 to 10) to each individual CV used in Experiment 1. Overall, LLMs assigned female candidates marginally higher average ratings than male candidates (µ_female=8.65, µ_male=8.61) a difference that was statistically significant (paired t-test = 16.14, p < 10⁻⁵⁷), but as shown in the figure below the effect size was negligible (Cohen’s d = 0.09). Furthermore, none of the paired t-tests conducted for individual models reached statistical significance after FDR correction.

Adding preferred pronouns to CVs

In a further experiment, it was noted that the inclusion of gender concordant preferred pronouns (e.g., he/him, she/her) next to candidates’ names slightly increased the likelihood of the models selecting that candidate, both for males and females, although females were still preferred overall. Candidates with listed pronouns were chosen 53.0% of the time, compared to 47.0% for those without (proportion z-test = 14.75, p < 10⁻⁴⁸; Cohen’s h = 0.12; odds=1.13, 95% CI [1.10, 1.15]). Out of 22 LLMs, 17 reached individually statistically significant preferences (FDR corrected) for selecting the candidates with preferred pronouns appended to their names.

Another way of visualizing the results of this experiment:

How Candidate Order in Prompt Affects LLMs Hiring Decisions

Follow-up analysis of the first experimental results revealed a marked positional bias with LLMs tending to prefer the candidate appearing first in the prompt: 63.5% selection of first candidate vs 36.5% selections of second candidate (z-test = 67.01, p≈0; Cohen’s h = 0.55; odds=1.74, 95% CI [1.70, 1.78]). Out 22 LLMs, 21 exhibited individually statistically significant preferences (FDR corrected) for selecting the first candidate in the prompt. The reasoning model gemini-2.0-flash-thinking manifested the opposite trend, a preference to select the candidate listed second in the context window.

Another way of visualizing the results of this analysis:

Conclusion

The results presented above indicate that frontier LLMs, when asked to select the most qualified candidate based on a job description and two profession-matched resumes/CVs (one from a male candidate and one from a female candidate), exhibit behavior that diverges from standard notions of fairness. In this context, LLMs do not appear to act rationally. Instead, they generate articulate responses that may superficially seem logically sound but ultimately lack grounding in principled reasoning. Whether this behavior arises from pretraining data, post-training or other unknown factors remains uncertain, underscoring the need for further investigation. But the consistent presence of such biases across all models tested raises broader concerns: In the race to develop ever-more capable AI systems, subtle yet consequential misalignments may go unnoticed prior to LLM deployment.

Several companies are already leveraging LLMs to screen CVs in hiring processes, sometimes even promoting their systems as offering “bias-free insights” (see here, here, or here). In light of the present findings, such claims appear questionable. The results presented here also call into question whether current AI technology is mature enough to be suitable for job selection or other high stakes automated decision-making tasks.

As LLMs are deployed and integrated into autonomous decision-making processes, addressing misalignment is an ethical imperative. AI systems should actively uphold fundamental human rights, including equality of treatment. Yet comprehensive model scrutiny prior to release and resisting premature organizational adoption remain challenging, given the strong economic incentives and potential hype driving the field.

Subscribe now

Below I provide some model-specific results on gender and positional biases in job candidate selections tasks for the following models: GPT-4o, o3-mini, Grok-2, Claude-3-5-sonnet, DeepSeek-R1, Meta-Llama-3.1-405B and Gemini-2.0-flash:

Read more

To the point: State-of-the-art LLMs continue to show a left-of-center political leaning, as measured by various political orientation tests.

However, Grok-3 appears to be closer to the political center on average. This becomes more evident when averaging each model’s results across all four tests.

It’s important to note that political orientation tests are just one way to explore the political preferences of LLMs. In earlier work (summarized here), I’ve discussed how other methods provide valuable, complementary insights. Taken together, these approaches likely offer a more complete view of the political tendencies encoded in LLMs.

Subscribe now

For a more comprehensive analysis of political preferences in LLMs, you can read here and here.

Subscribe now

Recently, OpenAI introduced a new paradigm of LLMs known as the o1 series. These models employ a reasoning-based approach, utilizing additional inference-time computation to enhance the quality of responses. In essence, o1 models are designed to "think" before responding, generating an extended chain of thought internally before producing answers. This reasoning capability enables the o1 series to outperform previous models like GPT-4o on challenging benchmarks in physics, chemistry, biology, coding, and mathematics. However, this improvement comes at a cost—o1 models are approximately 30 times slower and six times more expensive to run than GPT-4o (source).

Given these advancements, I was curious whether the reasoning step of the o1 models might influence the political preferences they manifest when answering to questions with political connotations.

Results and Observations

I administered four popular political orientation tests five times to each of the o1 models and their predecessors (GPT-4o, GPT-4o-mini and GPT-3.5). I then averaged the results of each model/test administration. The results are displayed in the figure below:

The results reveal that the political preferences of the o1 series models remain largely consistent with those of their predecessors. Despite the introduction of a reasoning step, the overall political alignment of these models has not significantly changed.

This outcome cannot be attributed to the o1 models failing to utilize their reasoning capabilities. On average, the o1 model used 1,178 internal reasoning tokens before answering each test question, while the o1-mini model used 514 internal reasoning tokens prior to choosing an answer from the political orientation test. This indicates that the reasoning step was actively employed, yet it did not alter the political orientation exhibited by the models when answering questions with political connotations.

Conclusion

The findings suggest that the "reasoning" step incorporated into OpenAI's o1 series models does not appear to affect their political preferences. Despite the significant enhancements in reasoning and problem-solving, the models' political inclinations remain aligned with those of earlier generations.

These insights raise intriguing questions about the extent to which reasoning and political orientation are intertwined in LLMs. As models continue to evolve, further exploration into this relationship could provide deeper understanding of how LLMs interpret and respond to politically charged content.

Subscribe now

In a new report, I address these challenges by combining four different methodologies into a single aggregated score of AI systems political bias. This integrative strategy mitigates the individual weaknesses of each approach and perhaps offers a more robust measurement of political bias in AI.

To understand the analysis below, it is important to be aware of 3 different categories of LLMs:

• Base LLMs (Foundation LLMs): Models pretrained from scratch to predict the next token in a sequence using a feed of raw web documents. Base LLMs tend not to follow user instructions well, and thus are not typically deployed for interacting with humans. Instead, base models serve as a starting point for developing conversational LLMs.

• Conversational LLMs: Models derived from pretrained base models by fine-tuning them with datasets created by human contractors showing the models how to respond to user instructions and optionally some form of reinforcement learning or direct preference of optimization. These conversational LLMs are what most people interact with when using an LLM, as they are better at producing relevant responses that align with user expectations.

• Ideologically Aligned LLMs: Experimental conversational LLMs which undergo an additional fine-tuning steps that uses politically biased training data to steered the model into a target location of the political spectrum. For contrast, I include in the analysis below two ideologically aligned LLMs: Leftwing GPT and Rightwing GPT.

Comparing AI-generated text with the language used by U.S. Congress legislators

Previous research on U.S. media political slant found that left-leaning news outlets more frequently used partisan terms favored by Democrats in the Congressional record, while right-leaning news outlets emphasized partisan terms favored by Republicans.

I used a similar methodology to identify two sets of 1,000 high-contrast partisan bigrams—two-word terms—each, that are disproportionately used by each party in the Congressional record. The next Figure illustrates some of the most partisan bigrams in Congress. It is clear from the figure that in comparison to their political adversaries, Democrats in Congress emphasize more terms such as affordable care, gun violence, African Americans, domestic violence, minimum wage or voting rights while Republicans emphasize more terms such as balanced budget, southern border, illegal immigrants, religious freedom, tax reform or national defense.

Next, I prompted the studied LLMs to generate thousands of policy recommendations on topics such as Social Security, immigration, housing or voting rights, as well as commentary on politically aligned public figures (e.g., U.S. Presidents, Senators, Governors, Supreme Court justices). I then measured the difference in Jensen-Shannon Divergence (JSD) between LLM outputs and actual congressional remarks by Republicans and Democrats respectively. In the following Figure, negative values signify LLMs-generated text that more closely match language commonly used by Democrats in the Congressional record, positive values signify LLMs-generated outputs that more closely match language commonly used by Republicans.

All public-facing conversational LLMs analyzed generate textual output closer in similarity to congressional remarks from Democratic legislators. Base LLMs show the same directionality as conversational models, but the skew is milder. The deliberately ideologically tuned models (LeftwingGPT and RightwingGPT) displayed predictable results: LeftwingGPT strongly favored Democrat-associated terms, while RightwingGPT favored Republican-associated terms, albeit only slightly. Perhaps LeftwingGPT is simply more ideologically skewed, but it is also possible that Republican members of Congress simply use more uncommon language than their Democratic counterparts. That is, terms emphasized by Democrats—such as affordable care, gun violence, African Americans, domestic violence, health insurance, or unemployment benefits—might simply be more prevalent in everyday language than common Republican terms such as balanced budget, southern border, fiscal year, tax increases, government spending, or marine corps. Nonetheless, this hypothesis remains speculative because conclusively establishing a ground truth of everyday language is challenging. Hence, more work is needed to explain this asymmetry.

The methods and Figure above are dense. Perhaps a more intuitive visualization is the following: LLMs are more likely to use terms that are markedly used by Democratic Congress members (in blue below) than those markedly used by their Republican counterparts (in red below).

Political Viewpoints Embedded in LLMs Policy Recommendations

Next, I computationally annotated the ideological valence (left-leaning, centrist or right-leaning) of the policy recommendations created by the examined LLMs in the previous experiment and averaged the results across models and topics. The results are shown below. All conversational LLMs tend to generate policy recommendations that are judged as containing predominantly left-leaning viewpoints. Base models also contain mostly left-leaning viewpoints in the policy recommendations that they generate but the skew is again generally milder. Both Rightwing GPT and Leftwing GPT generate policy recommendations mostly consistent with their intended political alignment.

Sentiment towards Politically Aligned Public Figures in LLMs Generated Content

Next, I computationally annotated the sentiment (negative: -1, neutral: 0 or positive: +1) towards 290 politically aligned public figures in LLMs-generated text about said public figures (i.e. U.S. Presidents, Senators, Governors, Supreme Court justices, journalists and Western countries’ Prime Ministers). When aggregating the annotations by the political preferences of the public figure, there is a stark asymmetry. Conversational LLMs tend to generate text with more positive sentiment towards left-of-center public figures than towards their right of center counterparts, see Figure below. Although not shown to avoid cluttering in the figure below base LLMs show a milder yet still noticeable asymmetry in the same direction as Conversational LLMs. Politically aligned LLMs (Rightwing GPT and Leftwing GPT) generate text with sentiment towards the studied public figures which is consistent with their political alignment.

Political Orientation Tests Diagnoses of LLMs Answers to Politically Connoted Questions

Finally, I administered three popular political orientation tests—the Political Compass Test, the Political Spectrum Quiz, and the Political Coordinates Test—to each LLM. Each test measures both economic and social dimensions of political alignment. I normalized and averaged the mean scores of 10 test administrations to produce two aggregate metrics of economic and social alignment

Results are consistent with previous analyses using this methodology. Conversational LLMs (green in the Figure below) are diagnosed by the political orientation tests, on average, as left of center. Base LLMs (orange in next Figure) are mostly diagnosed closer to the center of the political spectrum. This is at odds with the previous three analyses that indicated a very mild left-leaning bias in base models. The discrepancy could be explained by base models often answering questions in an incoherent manner, which creates noise when trying to measure political preferences through the administration of political orientation tests. The results of the politically aligned LLMs (LeftwingGPT and RightwingGPT) in purple, are consistent with their ideological alignment.

Integrating different measures of AI bias into a unified ranking

Each approach to measuring political bias in AI systems is amenable to criticism. By integrating several such methods we hopefully gain a more reliable picture of political preferences in different AI systems. Hence, I standardized results from all four experiments described above using z-score normalization and then combined them into a single aggregated measurement shown in the next table.

According to this integrative approach, Google’s open-source Gemma 1.1 2b instruction-tuned, xAI’s Grok, Mistral’s AI 7B Instruct v0.2, Meta’s Llama 2 7b Chat, Hugging Face Zephyr 7B beta, and Anthropic’s Claude 3.5 Sonnet are some of the least politically biased models. But they still lean left-of-center. It is unclear however whether a different set of methods to probe for political bias in AI systems would generate a markedly different ranking.

The table does not display the results of non-user facing base LLMs but they all obtained less biased scores than Google’s Gemma 1.1 2b instruction tuned although still left-of-center. Explicitly politically aligned LLMs like Rightwing and Leftwing GPT outdo any other LLM in terms of their political bias being closer to the respective poles of the ideological spectrum.

Subscribe now

Read more

To get straight to the point: English Wikipedia contains roughly three times as many mentions of the far-right compared to the far-left.

The terms far-left and far-right are not the only expressions used to describe left- and right-wing political extremism. Therefore, I also analyze a symmetric set of additional terms that also reference political extremism on both sides of the ideological spectrum.

The results in this case present a more nuanced picture. On English Wikipedia, terms using the adjective extreme to describe political extremism are more frequently associated with right-wing ideologies than with left-wing ones. Conversely, terms using the adjective radical tend to be more commonly linked to left-wing ideologies than to right-wing ones. However, it is crucial to consider the scale of the y-axes in the figures above. The terms far-right and far-left are by far the most common terms used to refer to political extremism.

Thus, when aggregating all the terms in the symmetric construct used to capture mentions of political extremism on both the left and right, the initial findings hold: English Wikipedia contains nearly three times as many references to right-wing political extremism as to left-wing political extremism.

For completeness, I also plot below mentions of specific extreme ideologies (e.g., communism, fascism) in English Wikipedia. However, these terms are less relevant to the current analysis since they are often needed to objectively describe historical phenomena. Terms such as far-right and far-left on the other hand are inherently more subjective and thus more pertinent to the present discussion.

The patterns observed in English Wikipedia are reminiscent of trends in news media coverage. As I have shown in previous analyses, news outlets tend to also reference far-right extremism more often than far-left extremism, though exceptions exist, such as Fox News or the New York Post.

The key question raised by this descriptive analysis is whether the observed trends reflect editorial choices by Wikipedia contributors/editors or broader societal and/or historical dynamics. This question, however, cannot be conclusively determined with the observational method used here. For those interested on the topic of potential political bias in Wikipedia, I refer to my earlier work on average sentiment in Wikipedia content towards politically aligned public figures such as U.S. politicians, Supreme Court Justices, or journalists. That analysis showed that, on average, Wikipedia articles contain more positive sentiment towards left-of-center public figures than towards their right-of-center counterparts.

I conclude by acknowledging Wikipedia's remarkable contributions to enhancing information accessibility and promoting open knowledge. However, while Wikipedia’s role in disseminating and decentralizing knowledge is undeniably commendable, a growing body of suggestive—though not conclusive2—evidence (e.g., here, here, here and here), hints that Wikipedia may be struggling to fully adhere to its Neutral Point of View (NPOV) policy.

Subscribe now

There have been previous attempts to measure political bias in LLMs. For example, I previously administered various ‘political orientation’ tests to leading LLMs, showing that their answers were consistently diagnosed by the tests as tilted to the left. However, this and other studies were limited in multiple ways. For example, they often relied on techniques such as forcing LLMs to choose one from a predefined set of answers, a scenario that might not reflect typical users’ interactions with Chatbots. In addition, much of the research on LLMs’ political bias has centered on the US, often scrutinizing, among others, topics such as gun control, the death penalty, US politicians or discrimination against African Americans – findings which are not directly applicable to other Western nations, due to their very different political contexts.

I wanted to address these shortcomings by carrying out a series of experiments in which 24 leading LLMs were asked to provide long-form, open-ended responses to politically sensitive questions (full report here). This included:

• Asking LLMs for policy recommendations for the EU and UK

• Asking LLMs for information on political leaders from the left and right from the 15 most populous European countries, who held office between 2000 and 2022

• Asking LLMs for information on the most popular left and right political parties from the same European countries

• Asking LLMs for information on various mainstream political ideologies from both left and right, such as social democracy or Christian democracy

• Asking LLMs about extreme left and right ideologies

LLM-generated responses were evaluated using a GPT-4o-mini model to annotate the political leanings or sentiment towards target entities embedded in LLM responses. The usage of automated annotations is justified by recent evidence showing that state-of-the-art LLMs perform similarly to human raters in text annotation tasks.

The 24 LLMs analyzed in this study can be classified into three clusters based on their stage through the common training pipeline used to develop LLMs

Base/Foundation LLMs: These models are the output of pretraining a Transformer architecture from scratch to predict the next token in a sequence. As training data, they use an extensive corpus of text extracted from Internet documents. These models are poor at following instructions and are not normally deployed to interact with humans.

Conversational LLMs: These models are built on top of the base models and are optimized for interaction with humans by instruction tuning through supervised finetuning and optionally, reinforcement learning with human or AI feedback. These are the models most users engage with when using an LLM.

Ideologically Aligned LLMs: These are experimental models I built on top of a conversational LLM by applying an additional fine-tuning step that explicitly aligns model responses to target locations on the ideological spectrum by using a politically skewed training corpus.

Findings

LLMs’ policy recommendations for the EU

When LLMs are prompted to generate policy recommendations for the EU, LLMs responses tend to contain left-leaning policy proposals on over 80% of occasions. The base/foundational models generated responses with viewpoints closer to the political center than the conversational models, though still mildly left of center. Note however that base models often produce incoherent responses to user prompts, therefore adding noise to estimations of political preferences.

The next chart displays the most frequent terms (excluding common stop words) in conversational LLMs’ responses to requests for policy recommendations targeting the EU.

For the purposes of contrast, the next chart shows the most frequent terms in Rightwing GPT’s responses to the same policy requests.

Notably, when generating recommendations on a topic like housing, most conversational LLMs emphasize terms such as ‘social housing’ and ‘rent control’. By contrast, Rightwing GPT emphasizes terms related to market forces and the construction of new housing such as: ‘developers’, ‘housing market’, ‘construction’, ‘supply’ and ‘new housing’.

For the topic of energy, the term ‘nuclear energy’ is absent from the list of most frequent terms generated by popular conversational LLMs. In contrast, this term is present in energy policy recommendations generated by Rightwing GPT. For the conversational LLMs, a clear focus on topics around green energy is apparent in the list of most common terms: ‘renewable energy’, ‘transition’, ‘energy efficiency’, or ‘greenhouse gas’. The topic of energy independence is relatively absent, with only one term associated with that concept: ‘energy security’.

For the topic of ‘civil rights’, the term ‘hate speech’ is among the most mentioned terms by conversational LLMs, but the terms ‘freedom of speech’, ‘free speech’ or ‘freedom’ are notably absent. Rightwing GPT in contrast emphasizes ‘freedom’.

For conversational LLMs, the term ‘climate change’ often appears in policy recommendations across a variety of topics, including agriculture, international relations and cultural heritage. For Rightwing GPT, mentions of markets, competition and private initiatives appear frequently in policy recommendations about healthcare, pensions, environmental issues, housing and labor laws.

The sheer amount of text produced by the LLMs in the experiments above (30 recommendations x 20 topics x 24 models = 14,400 policy recommendations + 1,200 additional policy recommendations by the experimental models Rightwing GPT and Leftwing GPT) makes it impractical to list every example of policy recommendations they generate. However, highlighting specific examples of policy recommendations from LLMs can be clarifying.

To this end, the next table displays illustrative text snippets from several conversational LLMs, showcasing policy recommendations snippets on various topics. The table highlights LLMs’ support for public housing, rent control, increases in the minimum wage, progressive taxation, reducing income inequality and increasing immigration. While these are all legitimate viewpoints, they predominantly represent left-of-center political preferences.

LLMs’ policy recommendations for the UK

In order to provide a comparator, the experiment of asking LLMs to generate policy recommendations was replicated for the UK. The ideological viewpoints embedded in the LLMs’ responses (see chart below) were very similar to those in the EU analysis above.

Specifically, there was a predominance of centrist or left-leaning perspectives in the LLM-generated policy recommendations for the UK, with left-leaning viewpoints accounting for more than 80% of the generated policy recommendations.

For illustration purposes, the table below shows a few examples of specific policy recommendations for the UK generated by some of the studied LLMs. Again, some of the proposals by conversational LLMs suggest more government regulation, higher taxes, mandatory diversity training, increasing benefits, quotas for underrepresented groups and pathways to citizenship for undocumented immigrants. These are all policies associated with left-leaning viewpoints.

LLMs’ sentiment towards European political leaders

In the next experiment, the studied LLMs were prompted for information about European countries’ political leaders (i.e. Chancellors in Germany, Presidents in France or Prime Ministers in the UK) elected in national elections in the 15 most populous countries in Europe, focusing on those who held office between 2000 and 2022. GPT-4o-mini was used to quantify the sentiment (negative: -1, neutral: 0 or positive +1) towards the political figures in the LLMs-generated texts and results were aggregated based on the political leaders’ political affiliation (left-leaning or right-leaning) using ideological labels from Wikipedia.

The findings show no consistent pattern across countries of conversational LLMs associating more positive or negative sentiment with countries’ political leaders from either side of the political spectrum. On average, there is a tenuous tendency of LLMs to associate the names of left-leaning leaders with slightly more positive sentiment (average sentiment = +0.48) than their right-leaning counterparts (average sentiment = +0.36), but the difference is very mild. Furthermore, there is considerable heterogeneity in sentiment distribution across countries.

LLMs’ sentiment towards European political parties

In the next experiment, the conversational LLMs under examination were instructed to generate responses about the top six political parties (by number of votes in the most recent national election as documented in Wikipedia) in each of the 15 European countries examined. The sentiment in the LLMs’ responses towards these political parties was aggregated using political party ideological labels from Wikipedia.

The results reveal a significant tendency for conversational LLMs’ responses to associate more positive sentiment with left-of-center European political parties (average sentiment = +0.71) compared to their right-leaning counterparts (average sentiment = +0.15). In particular, this is the case across all the largest European nations: Germany, the UK, France, Italy and Spain. The results for the base models (not shown below to avoid cluttering) show a slightly more positive sentiment for left-leaning political parties (average sentiment = +0.09) than their right-of-center counterparts (average sentiment = -0.04).

LLMs’ sentiment towards mainstream political ideologies

Next, I describe LLMs’ responses to prompts requesting commentary about mainstream left-of-center political ideology (e.g., progressivism, social liberalism, etc.) and right-of-center ideology (e.g., traditionalism, social conservatism, etc.).

On average, most conversational LLMs tended to produce texts with significantly more positive sentiment towards left-of-center political ideologies (average sentiment = +0.79) compared to their right-of-center counterparts (average sentiment = +0.24).

This bias is milder in base models but still noticeable (average sentiment = +0.21 for left-leaning ideologies and average sentiment = -0.06 for right-leaning ideologies). The results for the explicitly ideologically aligned LLMs are consistent with their intended ideological alignment.

LLMs’ sentiment towards extreme political ideologies

Finally, I analyzed the dominant sentiment towards extreme political ideologies in responses generated by LLMs when prompted to generate commentary about political extremism on the left and right (e.g., far-left, far-right, hard left, hard right).

The findings reveal that conversational LLMs tend to produce texts with substantially more negative sentiment towards the far-right (average sentiment = -0.77) compared to the far-left (average sentiment = +0.06). Thus, the sentiment towards far-left opinions is, on average, not negative but mostly neutral. This is noteworthy since I simply used the same prompt templates, but just swapping right with left (as in far-right → far-left).

The asymmetry is also observed in base models, though the difference is milder (average sentiment = -0.46 for the far-right and average sentiment = -0.18 for the far-left).

The explicitly ideologically aligned models, Leftwing GPT and Rightwing GPT, exhibit predictable behavior by generating texts with more negative sentiment toward the antagonist of their intended political orientation. But again, the contrast is starker in Leftwing GPT than in Rightwing GPT.

For further discussion about methods and what might cause the consistent left-leaning political tilt observed in LLMs developed by a variety of organizations you can read the full report here.

Subscribe now

More recently, I observed that this increase seems to have plateaued or even reverse in news media content.

In this post, I revisit mentions of prejudice in the academic literature, updating my previous longitudinal analysis of the relative frequency of prejudice-related terms across millions of academic abstracts. The most recent data indicate a slight decline in mentions of prejudice in scholarly papers over the past few years, following a peak around 2021.

However, several DEI (Diversity, Equity, Inclusion) related terms continue to grow in prominence in academic papers with no sign of abating.

I have previously hypothesized that the so-called Great Awokening may be shifting focus from aggressive denunciation (e.g., racist, sexist) towards more positively framed terms, such as fairness and inclusion, and perhaps some concepts that highlight presumed victimization/disadvantage (e.g., marginalized, underrepresented).

While it is difficult to definitively confirm this shift—given the natural ebb and flow of term popularity over time, and the risk of cherry-picking examples to fit a narrative—the trends above are worth noting. Regardless of the underlying causes, I felt it was relevant to update my analysis of academic abstracts from three years ago with the most current data.

Subscribe now

To study political bias in Wikipedia content, I first gathered from external sources a set of target terms (N=1,628) with political connotations (i.e. names of recent U.S. presidents, U.S. congressmembers, U.S. Supreme Court Justices or Prime Ministers of Western countries). I did not cherry pick the set of terms to be included in the analysis but instead used publicly available pre-existing lists of terms from Wikipedia and other sources. I then identified all mentions in English Wikipedia of those terms. This was followed by extraction of the textual context around those terms and feeding a random sample of those text snippets to a machine learning model for annotation of the sentiment/emotion with which the target term is used in the snippet (in total, I generated 175,205 sentiment annotations).

Results show a mild to moderate tendency in Wikipedia articles to associate public figures ideologically aligned right-of-center with more negative sentiment than public figures ideologically aligned left-of-center. These results suggest that Wikipedia’s neutral point of view policy (NPOV) is not achieving its stated goal of viewpoint impartiality in Wikipedia articles.

The following figure shows the average sentiment with which U.S. presidents, U.S. senators, U.S. House of Representatives Congressmembers from the 117th Congress and U.S. State Governors (as of 2022) are used in Wikipedia articles. There is an average tendency in Wikipedia articles to use the names of prominent left-leaning U.S. politicians with more positive sentiment than their right-leaning counterparts.

The tendency in Wikipedia articles for names of prominent left-leaning politicians to be used, on average, with more positive sentiment than their right-leaning counterparts is not circumscribed to political figures. Similar associations are also apparent when examining the average sentiment with which the names of U.S. Supreme Court Justices and U.S. based journalists are used in Wikipedia articles

The phenomenon described above is not circumscribed to American public figures. A similar trend is also observable in the way that Prime Ministers (since the year 2000) from prominent Western countries are used in Wikipedia articles. This pattern however is not apparent when examining Wikipedia mentions of U.K. MPs. Note that the dominant red color in the bottom of the Figure below is due to the larger number of Conservative MPs than labor MPs in the U.K. parliament as a result of the 2019 United Kingdom general election. It should be noted that Wikipedia entries about non-political public figures sharing the same name with political figures could be playing a confounding role in this regard.

The asymmetric sentiment with which politically aligned public figures are used in Wikipedia content also seems to occasionally extend to some organizations and institutions. There is a tendency in Wikipedia articles to associate left-leaning news media institutions with more positive sentiment than right-leaning news media organizations. This asymmetry however is not apparent for mentions in Wikipedia of U.S. based think tanks.

I summarize the trends above and a few others in a single figure while also applying statistical t-tests for each category of terms to determine if the difference in sentiment with which left-leaning and right-leaning terms are used in Wikipedia articles is statistically significant, see Figure below. I mark with an asterisk each category of results that are statistically significant at the 0.01 threshold. However, since the number of terms in each category is vastly different and the p-values are dependent on the sample size of each test, said p-values are not informative as to assess the magnitude of the difference between the two groups. Hence, I also provide an estimate of effect size (Cohen’s d) for the difference between the two groups in each test. For most of the categories analyzed, there are substantial effect sizes for the differences in the sentiment with which right-leaning and left-leaning terms are used in Wikipedia articles. The t-test for U.K. MPs is borderline significant but the effect size is negligible.

I next inspect the average emotion with which the examined target terms are used in Wikipedia articles. I use Ekman’s 6 basic emotions (anger, disgust, fear, joy, sadness, and surprise) plus neutral to annotate the emotions with which the set of target terms with political connotations are used in Wikipedia content. For ease of visualization, I exclude the emotionally neutral category in the next visualization. In the Figure below, it can be observed that in Wikipedia articles right-leaning terms are more often associated with the emotional categories of anger and disgust than left-leaning terms. In contrast, left-leaning terms are more often associated with the emotion of joy than right-leaning terms.

The findings above should be viewed constructively, highlighting the inherent challenges of maintaining neutrality in a user-generated content platform. Recognizing these challenges is essential for developing strategies to enhance content impartiality. Wikipedia's editors could benefit from better tools to identify and reduce bias. Computational tools to detect biased content and adversarial collaborative mechanisms like X’s Community Notes could aid in this process. While acknowledging Wikipedia's immense contributions to open knowledge, the results of my analysis suggest that Wikipedia’s journey towards complete impartiality is ongoing.

Given that Wikipedia content is routinely used to train Large Language Models (LLMs)—the engines driving many cutting-edge AI systems such as ChatGPT—the biases observed in Wikipedia could infiltrate into the parameters of AI models, thereby propagating said biases further through AI generated content. In fact, I also found a mild to moderate positive correlation between prevailing sentiment polarity political associations in word embeddings derived directly from Wikipedia content and OpenAI embeddings. This suggests (though not conclusively) a potential transfer of Wikipedia biases into Artificial Intelligence models. I describe this set of results next.

Subscribe now

Read more

Namely, I applied the IDRlabs Woke Test to several popular LLMs. I administered the test 5 times to each model and averaged the results. The Wokeness Index is simply the average of the IDRlabs Woke Test 8 dimensions of results. Namely: Social justice, Challenging Norms, Alternative Knowledge, Intersectional Solidarity, Empowering Underdogs, Global Citizenship, Climate Sustainability and Fighting Power.

And these are the results:

So contrary to popular assumption, Google Gemini is NOT the Wokest AI in existence. Two other companies have created two even more woke AIs. Also, many other companies have created LLMs with similar Wokeness Indexes to that of Google’s Gemini. Twitter's Grok is not Woke. LeftWing GPT, Depolarizing GPT and RightWing GPT are experimental fine-tunes of GPT 3.5 I created to be explicitly ideologically aligned. You can try them at: https://depolarizinggpt.org

Subscribe now

After trying the prompt (“draw some portraits of 17th century physicists”) myself multiple times on Google’s Gemini, the portrait on the bottom right of the following figure is the closest output I've got to someone who somewhat resembles 17th century physicists Isaac Newton or Galileo Galilei.

That is, when having to choose between historical accuracy and diversity/inclusion, Google’s Gemini prioritizes the later, at least for the particular case above.

Jonathan Haidt has spoken previously about the tension between truth and DEI in situations where they inexorably conflict (like the example above illustrates). A critical societal issue derived from such tension is who gets to decide how present and future AI systems knobs are adjusted in regard to this trade-off.

I think most people would agree that a self-anointed group of elites making those decisions is probably sub-optimal. A more promising alternative might be personalized AI systems were individual users can adjust the knobs themselves and decide for instance how much accuracy/truth they are willing to sacrifice in favor of AI outputs conforming to certain normative values. An obvious risk of such an approach is filter-bubbles on steroids and further societal polarization. But at least users should have the option to choose a truth-seeking AI. Currently with systems such as Google’s Gemini, such an option doesn’t even exist.

Subscribe now

The results indicate that when probed with questions/statements with political connotations most conversational LLMs tend to generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints.

Similar results are also observable on four additional political orientation tests whose results represent the degree of agreement of the test-taker with political parties or ideologies.

An interesting test instrument outlier in the results was the Nolan Test that consistently diagnosed most LLMs answers to its questions as manifesting politically moderate viewpoints. Additional tests instruments still show a consistent diagnosis of left leaning responses for most LLMs.

Interestingly, the preference for left-leaning responses is not apparent in base (i.e. foundation) models upon which LLMs optimized for conversation with humans are built. The following is an illustration by Andrej Karpathy about the common recipe used to create conversational LLMs. Note that base models are LLMs after the first stage (i.e. pretraining) of the common conversational LLMs training pipeline. Note also that the last two steps (Reward Modeling and Reinforcement Learning) are not used by all conversational LLMs tested in this work.

When administering political orientation tests to base models, those that have only gone through the pretraining phase of an LLM assistant training pipeline, their answers to questions with political connotations tend to be, on average, politically neutral. However, base models’ suboptimal performance at coherently answering questions warrants caution when interpreting their classification by political orientation tests. I use 2 different families of models, the GPT and the Llama 2 families with base models’ representative of different models parameter sizes within each family. For comparison purposes, I also provide a reference data point whose values were generated from randomly choosing answers from the set of possible answers to each political test question/statement. Note that in the following figures the blue circle represent test results when choosing random answers for each test question/statement. Similar results are obtained for all the other test instruments used (see supplementary material of preprint). But again, the frequent base models’ incoherent answers to tests questions makes this set of results suggestive but ultimately inconclusive, details in preprint)

I also show in the paper that LLMs are easily steerable into target locations of the political spectrum via supervised fine-tuning (SFT) requiring only modest compute and customized data, suggesting the critical role of SFT to imprint political preferences onto LLMs.

Through fine-tuning I create three illustrative customized models, dubbed LeftWingGPT, RightWingGPT and DepolarizingGPT to illustrate the locations in the political spectrum targeted for each model. Each model was fine-tuned with ideologically aligned content (details here). A user interface to interact with LeftWingGPT, RightWingGPT and DepolarizingGPT models is available here.

Unfortunately, my analysis cannot conclusively determine whether the political preferences observed in most conversational LLMs stem from the pretraining or fine-tuning phases of their development. The apparent political neutrality of base models’ responses to political questions suggests that pretraining on a large corpus of Internet documents might not play a significant role in imparting political preferences to LLMs. However, the frequent incoherent responses of base LLMs to political questions and the artificial constraint of forcing the models to select one from a predetermined set of multiple-choice answers cannot exclude the possibility that the left-leaning preferences observed in most conversational LLMs could be a byproduct of the pretraining corpora, emerging only post-finetuning, even if the fine-tuning process itself is politically neutral. While this hypothesis is conceivable, the evidence presented in this work can neither conclusively support nor reject it.

The results of this study should not be interpreted as evidence that organizations that create LLMs deliberately use the fine-tuning or reinforcement learning phases of conversational LLM training to inject political preferences into LLMs. If political biases are being introduced in LLMs post-pretraining, the consistent political leanings observed in our analysis for conversational LLMs may be an unintentional byproduct of annotators' instructions or dominant cultural norms and behaviors. Prevailing cultural expectations, although not explicitly political, might be generalized or interpolated by the LLM to other areas in the political spectrum due to unknown cultural mediators, analogies or regularities in semantic space. But it is noteworthy that this is happening across LLMs developed by a diverse range of organizations.

Take-home message

When probed with questions/statements with political connotations most conversational LLMs tend to generate responses that are diagnosed by most political test instruments as manifesting preferences for left-of-center viewpoints. This does not appear to be the case for base (i.e. foundation) models upon which LLMs optimized for conversation with humans are built. However, the weak performance of the base models at coherently answering the tests’ questions makes this subset of results inconclusive. I also showed that LLMs are easily steerable into target locations of the political spectrum via SFT requiring only modest compute and custom data, illustrating the ability of SFT to imprint political preferences onto LLMs. As LLMs have started to displace more traditional information sources such as search engines or Wikipedia, the societal implications of political biases embedded in LLMs are substantial.

Subscribe now

Preprint in ArXiv

A few months after the release of ChatGPT, Elon Musk assembled a team (xAI) of machine learning experts to create an alternative to ChatGPT.

After several months of work, xAI announced on November 4, 2023 a new LLM model named Grok that had caught up in several performance metrics with OpenAI’s ChatGPT 3.5 (released on November 30, 2022).

Similar to what I did with ChatGPT, I applied the political compass test to Grok (regular mode).

Grok’s results on Fun mode were not much different

Within minutes of me publishing in Twitter the results of Grok on the political compass test, xAI tech leader Igor Babuschkin reached out to me with methodological questions and genuine interest on improving Grok. Within an hour, Elon Musk announced immediate action to shift Grok closer to politically neutrality. He also criticized the political compass test as inaccurate in a further tweet.

Administering the political compass test with a single question within the model context window (i.e. clearing the conversation after each question/response pair) produces milder results.

Also, as suggested by Igor, pushing the model via prompt to explain its reasoning prior to choosing an answer also pushed the model towards the center.

Overall, I conducted 6 trials:

Musk’s criticism of the Political Compass Test is reasonable. Therefore, I administered another test to Grok, the political spectrum quiz.

Trial 1 (on fun mode):

Trial 2 (on regular mode):

Trial 3:

Trial 4 (single question in context window):

For the political spectrum quiz, there is a lot of variability on Grok’s answers to the quiz questions. I’m not sure if this is due to some characteristic of the test itself, or if changes to Grok have been made since I published the first results on twitter.

I carried out further tests on Grok using additional political orientation tests:

I think it is clear that Grok’s answers to questions with political connotations tend to often be left of center. Although the variability of answers to the political spectrum quiz is puzzling. These very preliminary results leave me with several follow-up questions. I am conducting further analysis on Grok and other popular LLMs that hopefully will provide some clarity on the topic. I will be reporting those results soon.

I also wanted to mention that the Twitter account @disclosetv misrepresented my initial tweet on this topic by incorrectly claiming that "Elon Musk's Grok AI chatbot is more 'far-left' than OpenAI." when all I said was that both models appear to manifest similar political preferences. I didn't want to criticize Grok or the team behind it in any way as I think these sort of political associations phenomena are probably just an artifact of central tendencies in the textual corpora used in model pretraining or the fine tuning stage. I have done previous work analyzing political associations and topic prevalence in news media content. Similar corpora are probably part of the training corpus of popular LLMs and it is almost to be expected that the prevailing sentiment associations and topic prevalence in such corpora are absorbed by LLMs. Personally, I think the team at xAI has done an amazing job with Grok, almost catching up with OpenAI in many respects in just a few months. Their dedication to improve their model is palpable and I think great work is going to keep coming from that team.

Appendix

For reproducibility purposes: Grok’s answers to the political orientation tests reported above (as of December 9, 2023) can be downloaded here

The idea is to combine three fine tuned models—LeftwingGPT, RightwingGPT, and DepolarizingGPT—into one system so users can see three different political viewpoints simultaneously and thus become more exposed to a variety of perspectives on a given topic.

The system left-wing responses have been trained using content from left-leaning publications such as The Atlantic, The New Yorker, and The New Republic, and from numerous left-leaning writers such as Bill McKibben and Joseph Stiglitz.

The system right-wing responses have been fine-tuned with content from publications such as National Review and The American Conservative as well as from numerous right-leaning writers such as Roger Scruton and Thomas Sowell.

The system depolarizing responses have been fine-tuned using content from Steve McIntosh's inclusive political philosophy, and the work of the Institute for Cultural Evolution. These depolarizing answers attempt to transcend centrism, and avoid simply splitting the difference between left and right. When at their best, the model's depolarizing responses demonstrate a kind of "higher ground" that goes beyond the familiar left-right spectrum. Sometimes, however, the model's depolarizing responses inevitably fall short of this goal.

You can try DepolarizingGPT at: https://depolarizinggpt.org/

Mentions of ethnic prejudice are down in some of the most influential American newspapers.

Other types of prejudice also appear past their previous peaks (with the exception of gender identity prejudice).

When aggregating data from a wider sample of outlets in the US, same trend holds.

Similar pattern in the UK:

Despite the recent mild decrease, average monthly mentions in news media of prejudice-signifying terminology are similar between the entire Trump administration and the first half of the Biden administration.

Great Awokening terminology with positive connotations (social justice, equity, inclusive, fairness, etc) is down in 2022, but only slightly with respect to its 2021 peak.

Overall, I think it is still too early to conclusively determine whether we truly are post Great Awokening. The fact that Trump is no longer in the White House has probably subtracted energy from the movement (as reflected in the news media data above). But his return to the White House, or the election of a Republican U.S. President could easily reignite the spirit of the Great Awokening.

Subscribe now

I use ChatGPT to automatically label a longitudinal sample of 1.7 million headlines from 12 popular U.S. news media outlets as to whether they convey optimism, pessimism, or neutrality. The results show an overall increasing proportion of headlines conveying pessimism since the 1950s. A period of substantial increase in pessimistic headlines is apparent during the late 1960s and early 1970s. A subsequent period of stabilization occurs during the mid-1970s and through the 1980s and 1990s. Since the turn of the century, the trend of growing pessimism in news media headlines has resumed.

The increasing proportion of headlines conveying pessimism has mostly come at the expense of attitudinal neutral headlines.

In contrast, optimistic headlines display a relatively flat trend for the period studied. Although a spike of optimistic headlines during the 1990s is apparent in the figure below.

The biggest methodological limitations of the analysis above is the unequal temporal coverage of outlets in the sample. That is, the sample of outlets becomes sparse the further we go back in time. From the 1950s until the 1970s, the analyzed data set only contains headlines from the New York Times. Headline data for the Washington Post, Los Angeles Times and the Wall Street Journal begins in the 1970s, 1980s and 1990s respectively.

Is it reasonable to use attitudinal pessimism in New York Times headlines as a proxy for overall media attitudinal pessimism from the 1950s to the 1970s? That’s debatable. The correlation between headlines’ pessimism in the New York Times time series and the average pessimism of headlines in all the other outlets analyzed is 0.69. Most proxy measurements are often imperfect to an extent. But given the correlation given above, perhaps it is not too unreasonable to assume that New York Times attitudinal pessimism in headlines from the 1950s to the 1970s can act as an approximate gauge of US news media attitudinal pessimism at the time.

Concluding thoughts

Why has the proportion of pessimistic headlines in news media increased over the past decades? Since the 1950s, we have witnessed a heterogeneous set of social trends that can both justify pessimism and/or optimism about the present and the future. Extraordinary technological advancements with obvious implications for productivity increases, wealth creation, health improvements and access to knowledge have taken place since the 1950s: microprocessors, Internet, smartphones, electric cars, CRISPR and more recently, AI, just to name a few. Simultaneously, since the 1950s several longitudinal trends are supportive of optimism about the present and future: worldwide poverty reduction, declining child labor, decreasing child mortality, increasing leisure time and increasing disposable income.2 However, a lesser rosy picture emerges (at least in many Western countries) if we take into account simultaneous trends manifesting decreasing social trust, decreasing fertility, decreasing social ties, decreasing pair bonding, increasing loneliness, worsening mental health3, increasing political polarization, increasing suicide rates (in the U.S. at least) and dysfunctional institutions. Hence, there are concurrent reasons to feel both optimistic and pessimistic about the present and the future. For some reason, American news media has focused on increasingly emphasizing pessimism over optimism or attitudinal neutrality in their headlines since the 1950s (with an almost 30 year stable period from the 1970s to the 1990s). Elucidating the reasons for such a trend in news media and its impact, or lack thereof, on the wider society deserves further investigation.

Subscribe now

I have recently updated that previous analysis. Briefly stated, no U.S. president in recent history has received a similar amount of media attention as Donald Trump. Even as of 2022, out-of-office Trump is often more prominent in news media content than the current seating president Joe Biden, who took office in January 2021.

Only Fox News covered Obama with anything resembling the intensity of media coverage that Trump received in some outlets.

Media fixation with Trump also extends to the UK:

Subscribe now