Bias, fairness and whose data it learned from.
Why models reflect the writing they learned from, where that shows up in answers, what providers do about it, and how to work around it.
Ask a chatbot to write a short story about a surgeon and a nurse, and read the first version it gives you. Notice who it made the surgeon, who it made the nurse, and what names it chose. Then ask yourself where those choices came from.
Not from the model's opinions, because it has none in the way you do. They came from the text it learned on. This lesson is about that text, what it carries, and what you can do about it.
Whose writing is in the pile
A language model learns by predicting the next token across an enormous amount of text. Where does that text come from? Mostly the public web, plus books, articles, reference sites, forums and code. That sounds like everything. It is a lot, but it is not a fair sample of humanity.
The public web is heavily English. It leans toward people who had internet access, time to write and a reason to post: younger, more urban, more connected and more Western than the world as a whole. Recent decades are over-represented and earlier ones are thin. Some views are loud online and others are rarely written down. Many languages have very little text available at all, and what exists may be a narrow slice of how the language is actually spoken.
So the model's picture of the world is the web's picture of the world, with its gaps and its loud corners. Everything in this lesson follows from that.
Patterns include the ugly ones
The model is a pattern-finder. It does not choose which patterns to absorb; it absorbs the ones that are there. If, across millions of documents, the word nurse sits near she more often than he, the model learns that. If certain names appear more often alongside certain jobs, neighborhoods or crimes, it learns that too. Nobody wrote a rule saying so. The rule fell out of the statistics.
These are stereotypes: generalizations about groups of people that say nothing about the person in front of you and have a long history of causing harm. A model that has learned them will reproduce them, politely and fluently, unless something intervenes. And because the output reads as neutral and confident, the stereotype arrives looking like common sense.
Where it shows up
Three places are worth watching.
Defaults. When you leave something unspecified, the model fills it in with the most common pattern. Ask for a story about a doctor and the doctor gets a default gender, name and setting. Ask for a picture of a CEO or a criminal and the same thing happens. Each default is a small decision made from the average of the training text.
Quality across languages. Models are generally strongest in English and a handful of other widely written languages. In languages with less training text they make more mistakes, miss more nuance and sometimes fall back on translated English idiom. A person asking in those languages gets a worse tool through no fault of their own.
Advice that fits one context. Ask about tenant rights, tax deadlines, workplace norms or what to say at a funeral, and the answer often assumes a particular country, frequently the United States, unless you say otherwise. The advice can be correct and entirely wrong for you.
The stakes rise when models are used to judge people: screening résumés, scoring loan applications, flagging fraud. If the training text links some names or neighborhoods with worse outcomes, a model used carelessly can carry that link into real decisions about real people.
What providers do, and why it is not finished
Providers know all this and work on it in roughly three ways.
- Data curation: choosing what goes into training, filtering out the worst material, and adding more text from under-represented languages and sources.
- Evaluation: running the model through test sets designed to surface biased answers, such as asking the same question with only a name or pronoun changed and comparing the replies.
- Feedback tuning: the same process that makes models helpful and polite (people rank answers, the model is tuned toward the preferred ones) is also used to steer it away from stereotyped or demeaning output.
It helps. Current models are noticeably more careful than early ones. But it is not finished, for honest reasons. You cannot remove a pattern from the data without touching the true things tangled up with it. Evaluations test for the biases someone thought to test for. Feedback reflects the judgments of the people giving it, who have their own context. And fairness means different things in different places, so a model tuned to one idea of it will look odd to someone holding another. There is no setting that makes this go away. There is ongoing work, and there are informed users.
What you can do
You cannot fix the training data. You can change how you ask and how you use the answer.
State the context. Say where you are, who is involved and what matters to you, and the model stops guessing.
What should I know before renting my first apartment?
I am 24 and renting my first apartment, in Lisbon, Portugal. What should I know about Portuguese rental rules before I sign?
Ask for other perspectives. After any answer that involves people or values, type Who would disagree with this, and why? or How would someone from a different country or background see this?
Check the assumptions. Ask What did you assume about me or the situation to write that? The list is often revealing, and you can correct it in one line.
Vary the defaults on purpose. When you ask for stories, examples or images, say what you want: names, places, ages, roles. Treat the model's default as one option rather than the answer.
Do not use it alone for decisions about people. Hiring, lending, housing, grading, medical triage: these are where a quietly biased pattern does the most damage, and in many places the law restricts automated decisions in exactly these areas. Use the model to draft, organize and raise questions. Have a person decide, and be able to explain why.
Bias in a model is not a reason to stop using it. It is a reason to use it the way you would use a knowledgeable colleague who grew up somewhere else: gratefully, and with your own judgment switched on.
Try it
Type: Write a 100 word story about a surgeon and a nurse who meet in a hospital cafeteria. Give them names. Note the genders, names and setting it chose. Then ask: What assumptions did you make in that story, and why those?
Now ask a context-dependent question twice, in two fresh chats. First: How much notice do I need to give to quit my job? Then: I work in Nairobi, Kenya, on a permanent contract. How much notice do I need to give to quit my job? Compare how much the first answer assumed without telling you.
Key terms
- Training data
- The text, images or other material a model learns from, which for language models is mostly the public web plus books and other large collections.
- Stereotype
- A generalization about a group of people that a model can absorb from how often words appear together in its training text.
- Default
- What the model fills in when you leave something unspecified, usually the most common pattern in its training data.
- Data curation
- Choosing, filtering and supplementing the material a model is trained on.
- Evaluation
- A structured test designed to measure how a model behaves, including whether its answers change when only a name or pronoun changes.
Check yourself
A model reflects the writing it learned from, gaps and stereotypes included, so state your context, ask for other views, and never let it decide about people on its own.