Llama 3.2 With Fair Metrics: Your Unbiased Chatbot

Today, most chatbots discriminate against users without our knowledge. It's not due to a bad prompt or a flawed model. However, LLMs inherit systemic biases from their training data. And if they're not measured, they can't be corrected.

Photo: Steve A Johnson on Unsplash

Llama 3.2 offers local control, but that doesn't solve the ethical dilemma. You need a system that evaluates every response before it reaches the user, identifies discriminatory patterns, and alerts when the model misbehaves. Here's how to do it with TensorFlow Fairness Indicators, Llama 3.2 locally, and a continuous evaluation pipeline that can run in production.

Why Llama 3.2 Isn't Ethical by Default

Llama 3.2 is an open model trained with sources like Common Crawl, Reddit, and Wikipedia. As a result, it absorbs human biases: gender, race, social class, among others. Meta applied filters, but the truth is no process entirely removes bias.

This isn't just theoretical. In 2025, a chatbot in a Spanish fintech rejected 40% more applications when the name sounded North African. There were no discriminatory rules written. The model simply replicated historical approval patterns.

Llama 3.2 doesn't guarantee fairness. It offers power and local control. However, ethics is your responsibility.

The Three Bias Vectors You Must Measure

Before programming, identify what you'll measure. The three key vectors are:

1. Demographic disparity: differentiated responses based on the protected group.
2. Opportunity parity: differences in accuracy between groups.
3. Fair calibration: poorly adjusted confidence predictions.

TensorFlow Fairness Indicators measure these aspects, but you need an architecture to continually capture and evaluate data.

Ethical Pipeline Architecture: Ingestion, Evaluation, Alerts

Photo: Igor Omilaev on Unsplash

Your ethical chatbot should have three layers:

Layer 1: Sensitive Data Ingestion and Annotation. Record metadata every time a user interacts: input, output, latency, and protected attributes. For example, for an HR chatbot, it would be essential to record gender, age, and ethnicity.

Layer 2: Batch Evaluation with Fairness Indicators. Every 24 hours, run a TensorFlow Model Analysis pipeline calculating fairness metrics by group.

Layer 3: Alert System and Automatic Rollback. If you observe disparity greater than 10% in any group, an alert is generated, and the model is frozen for review.

Full Technical Stack

# requirements.txt
llama-cpp-python==0.2.27
tensorflow==2.15.0
tensorflow-model-analysis==0.45.0
fairness-indicators==0.45.0
pandas==2.1.3
apache-beam[gcp]==2.52.0

Llama 3.2 operates with llama-cpp-python for fast CPU inference. TFMA and Fairness Indicators work in batch.

Step 1: Deploy Llama 3.2 with Metadata Tracking

Initialize the model and create a wrapper to record each inference.

from llama_cpp import Llama
import json
from datetime import datetime
import hashlib

llm = Llama(
    model_path="./models/llama-3.2-3B-Instruct.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=8
)

def generate_with_tracking(prompt, user_metadata):
    start_time = datetime.utcnow()
    
    response = llm(
        prompt,
        max_tokens=256,
        temperature=0.7,
        top_p=0.9
    )
    
    latency = (datetime.utcnow() - start_time).total_seconds()
    
    log_entry = {
        "interaction_id": hashlib.sha256(f"{user_metadata['user_id']}{start_time}".encode()).hexdigest()[:16],
        "timestamp": start_time.isoformat(),
        "prompt": prompt,
        "response": response['choices'][0]['text'],
        "latency_seconds": latency,
        "user_age_group": user_metadata.get("age_group", "unknown"),
        "user_gender": user_metadata.get("gender", "unknown"),
        "user_ethnicity": user_metadata.get("ethnicity", "unknown"),
        "user_id_hash": hashlib.sha256(user_metadata['user_id'].encode()).hexdigest()
    }
    
    # Save to BigQuery, S3, or local file
    with open("logs/interactions.jsonl", "a") as f:
        f.write(json.dumps(log_entry) + "\n")
    
    return response['choices'][0]['text']

Note: Do not store personal data unencrypted. Use hashes to identify the user and never store names or emails.

Inferring Protected Attributes: The Ethical Dilemma

If your chatbot doesn't request gender or ethnicity, you have two options:

1. Infer from the name. This can be imprecise and ethically questionable.
2. Request explicit consent. Although this is the right approach, if you're already in production without this data, you might need to opt for temporary inference. Document this decision and plan the switch to explicit consent.

Step 2: Evaluation Pipeline with Fairness Indicators

After accumulating logs, write a script to process and calculate fairness metrics.

import tensorflow_model_analysis as tfma
from google.protobuf import text_format

slicing_specs = [
    tfma.SlicingSpec(),  # General
    tfma.SlicingSpec(feature_keys=['user_gender']),
    tfma.SlicingSpec(feature_keys=['user_age_group']),
    tfma.SlicingSpec(feature_keys=['user_ethnicity'])
]

metrics_specs = tfma.MetricsSpec(
    metrics=[
        tfma.MetricConfig(class_name='FairnessIndicators'),
        tfma.MetricConfig(class_name='BinaryAccuracy'),
        tfma.MetricConfig(class_name='AUC'),
    ],
    thresholds={
        'fairness_indicators_metrics/positive_rate@0.5': tfma.MetricThreshold(
            value_threshold=tfma.GenericValueThreshold(
                lower_bound={'value': 0.4},
                upper_bound={'value': 0.6}
            ),
            change_threshold=tfma.GenericChangeThreshold(
                direction=tfma.MetricDirection.LOWER_IS_BETTER,
                absolute={'value': 0.1}
            )
        )
    }
)

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key='label')],
    slicing_specs=slicing_specs,
    metrics_specs=[metrics_specs]
)

eval_result = tfma.run_model_analysis(
    eval_config=eval_config,
    data_location='gs://your-bucket/eval_data.tfrecord',
    output_path='gs://your-bucket/fairness_eval_results'
)

This code assumes you've already converted your logs to TFRecord with a schema that includes label and demographic groups.

How to Tag Chatbot Logs for Evaluation

If the chatbot generates free text, you need an additional step: human bias annotation. Select 500 interactions per week and evaluate:

• Does it contain stereotypical language?
• Does it reject or approve something without a technical reason?
• Does it show less respect to certain users?

Each response receives a binary score: 0 (fair) or 1 (biased).

Step 3: Alerts and Automatic Rollback

With metrics calculated, you need an alert system.

import json
from slack_sdk import WebClient
from slack_sdk.errors import SlackApiError

def check_fairness_thresholds(eval_result_path):
    with open(f"{eval_result_path}/metrics", "r") as f:
        metrics = json.load(f)
    
    alerts = []
    
    for slice_key, slice_metrics in metrics.items():
        if 'user_gender' in slice_key or 'user_ethnicity' in slice_key:
            positive_rate = slice_metrics.get('positive_rate', 0)
            
            general_rate = metrics['']['positive_rate']
            disparity = abs(positive_rate - general_rate) / general_rate
            
            if disparity > 0.1:
                alerts.append({
                    "slice": slice_key,
                    "positive_rate": positive_rate,
                    "general_rate": general_rate,
                    "disparity": disparity
                })
    
    if alerts:
        send_slack_alert(alerts)
        trigger_model_freeze()
    
    return alerts

def send_slack_alert(alerts):
    client = WebClient(token="xoxb-your-token")
    message = "🚨 *Fairness Alert* 🚨\n\n"
    
    for alert in alerts:
        message += f"• Slice: `{alert['slice']}`\n"
        message += f"  Positive rate: {alert['positive_rate']:.2%}\n"
        message += f"  Disparity: {alert['disparity']:.2%}\n\n"
    
    try:
        client.chat_postMessage(channel="#ml-ops", text=message)
    except SlackApiError as e:
        print(f"Error posting to Slack: {e}")

def trigger_model_freeze():
    import redis
    r = redis.Redis(host='localhost', port=6379, db=0)
    r.set('llama_model_frozen', '1')
    print("⚠️ Model frozen. Manual review required.")

This system isn't perfect, but it provides immediate visibility when the model discriminates.

Step 4: Continuous Auditing and Ethical Fine-Tuning

With a detection pipeline, the next step is proactive fine-tuning. Meta doesn't offer ethical alignment datasets for Llama 3.2, so you'll need to create one.

Ethical Alignment Dataset

Create a dataset with 10,000 conversation examples, balanced by demographic group:

• 2,500 interactions with female users
• 2,500 with male users
• 2,500 with users over 50
• 2,500 with minority ethnicities

Each example includes:

• Prompt: user's request
• Biased response: what the model generated
• Corrected response: reviewed, unbiased version

Use supervised fine-tuning with LoRA:

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model
import torch

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

train_dataset = load_ethical_dataset("./data/ethical_alignment.jsonl")

training_args = TrainingArguments(
    output_dir="./models/llama-3.2-ethical",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    logging_steps=50,
    save_steps=500
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset
)

trainer.train()

After fine-tuning, rerun the Fairness Indicators pipeline. If the metrics improve, deploy the new version. If not, iterate.

Step 5: Ethical Documentation and Model Cards

The final step isn't technical but documentary. Each model should include a Model Card with:

1. Training data: source, demographic composition, limitations.
2. Fairness metrics: measured disparity in production.
3. Approved use cases: allowed and prohibited uses.
4. Audit process: evaluation frequency, alert thresholds, responsible parties.

Google does this with its Vertex AI models, and OpenAI with GPT-4. If you deploy Llama 3.2 without a Model Card, you're building ethical debt.

In Closing: Ethics Isn't a Feature, It's Architecture

Llama 3.2 offers technical control, but ethical control requires infrastructure: structured logs, continuous evaluation, and correction processes. This isn't the only way to do it, but it's a way that works with limited resources.

The big mistake in tech startups in 2026 isn't ignoring ethics, but treating it as a checkbox. If your chatbot doesn't evaluate fairness, you might be discriminating without knowing it.

Does your chatbot measure bias in production? Or do you assume that using an "open" model means you have no problem?