BMS 1110 · Common Core (Tronc commun)Section 6.3 · Sampling, estimation and inference

Section 29 of 30

Bias and survey design

Explain how sampling bias makes a sample systematically unrepresentative, name the bias in a described recruitment or survey method, and list the design decisions — from target population to response rate — that keep a survey honest.

New to this course? How each section works

Learn gives you one short idea at a time. Each idea starts with a single sentence, then explains it, gives an example, and asks you a small question you can answer straight away.
Explore shows a diagram of the same ideas. Click the numbered parts to read what each one is.
Check asks a few questions. Answer them all correctly to finish the section and unlock the next one.
“Go deeper” is optional. It adds mechanism and extra detail for when you are ready — nothing you need for the checks is hidden there.

Why this matters

Every number a hospital reports about its patients comes from somebody it chose to measure and from somebody who agreed to be measured. When those two groups are not like the whole population, the number is wrong in a fixed direction, and no amount of extra data repairs it. Bias is the reason two honest studies can disagree for years and the reason one staff survey can be read two opposite ways. Knowing its three common shapes is what lets you ask the right question of any figure put in front of you: not only “how big was the sample?” but “who was left out, and which way does that push the answer?”

What you need to know first

You need the vocabulary of sections 6.1 and 6.2: population, sample, parameter, statistic, and the idea that a simple random sample gives every member of the list the same chance of being chosen. All the arithmetic here is dividing, multiplying and reading percentages from unit 1.4, so keep a calculator if it helps; the numbers are small and every step is written out.

Words to know (6)

Sampling bias: A fault in the method used to choose a sample that systematically produces a group which does not adequately represent the target population, pushing every result in the same direction. Recruiting adults for an obesity study only at a fitness centre gives a sample that is not the adult population of the city.
Target population: The complete group the survey is meant to describe, fixed by clear rules of who, where and when — every member of it, not only the people who can be reached. For a ward survey of pain after surgery, the target population is every adult operated on in that ward during the study months.
Sampling frame: The real list from which names are drawn — a register, an address file, a payroll. Anyone missing from the list has no chance of being selected, however random the draw looks. Drawing from the voter register leaves out adults who are not registered, so the frame, not the draw, decides who can enter.
Survey: A systematic method of collecting information from individuals, in which the same questions are asked in the same order under the same rules for everyone. A hospital patient-experience survey asks every discharged patient the same twelve questions by the same route.
Poll: A survey designed to measure opinions, preferences or characteristics of a population. The word describes the purpose of the survey, not a different technique. A poll of nurses’ preferred shift pattern measures opinion; the same mailing could measure a fact such as how many were vaccinated.
Response rate: The number of usable answers divided by the number of people asked, usually written as a percentage. A low rate warns that the answers may not describe the whole group. If 480 of 1,000 mailed questionnaires come back, the response rate is 480 ÷ 1,000 = 0.48 = 48 %.

One idea at a timeIdea 1 / 6

In one sentence Sampling bias occurs when the method of choosing a sample systematically produces a group that does not represent the target population.

Sampling bias is a fault in the method, not bad luck

Sampling bias is present when the method used to select a sample systematically produces a sample that does not adequately represent the target population, and the notes add the consequence: it can lead to misleading conclusions. The decisive word is systematically. A random wobble pushes one sample a little high and the next a little low, so across many samples the wobbles partly cancel. A biased method pushes every sample the same way, so repeating the study reproduces the same distortion instead of correcting it. Bias is therefore a property of the procedure, not of the particular people who happened to answer. That is why bias is described by its direction — too high, too low, or unknown — and not only by its size, and why two carefully conducted studies can still disagree for years. The notes make the mechanism concrete: a researcher who wants the prevalence of obesity among all adults in a city but recruits only from a fitness centre gets a sample that may not represent the adult population, because people who attend a fitness centre differ from people who do not.

For example

People who attend a fitness centre tend to exercise more and weigh less than the city average, so an obesity estimate built on that sample is pushed downwards every time the study is repeated.

Pause and check yourself

A hospital measures hand-hygiene compliance only by watching staff who know that the auditor is present. What is the target population, and in which direction does the method push the result?

Show the answer

The target population is every hand-hygiene moment in the hospital. The method can observe only moments when staff know they are watched, and people tend to comply more when observed, so the estimate is pushed upwards systematically rather than at random. It is a fault of the method, and repeating the audit does not remove it.

Read detailed explanations

Bias has a direction and a size, and only one of them shrinks: Every estimate misses the truth for two reasons, and separating them is the whole skill of this section. Random error is the scatter produced by having measured only part of the population; its typical size is the standard error, which for a mean is s ÷ √n and therefore falls as the sample grows. Bias is the systematic displacement of the estimate away from the truth. The two combine in one quantity called the mean squared error, which is the variance of the estimate plus the square of the bias. Read the rule with numbers. A bias of 3 mmHg and a standard error of 1 mmHg give a mean squared error of 3² + 1² = 9 + 1 = 10, so the bias contributes nine tenths of the total. Now quadruple the sample: the standard error falls to 0.5 and the mean squared error becomes 9 + 0.25 = 9.25. Four times the data removed three quarters of the random part and almost none of the total error. That single calculation is why the direction of a bias matters more than its size. A two-sided error can be reduced by collecting more, while a one-sided error is permanent and becomes more convincing as the sample grows, because the interval tightens around the wrong value. Direction is also what makes a bias either repairable or merely reportable. If the city's true obesity prevalence is 24 % and a fitness-centre sample reports 18 %, the bias is 18 − 24 = −6 percentage points, that is, six points too low; knowing the sign allows a reader to correct the statement or at least to state the limit of the finding. A bias whose direction is unknown is worse, because the estimate could lie on either side and neither adjustment nor a cautious sentence is possible. In a real survey the truth is unknown, which is exactly why the direction must be reasoned from the method before any data arrive: who was excluded, and what are those people likely to have answered?
The three biases at work: which gate fails, and how to test it: Sampling bias enters through one of three gates, and naming the gate tells you which repair is available. The first gate is the frame, the list from which names are drawn. If the frame does not cover the target population, the fault is called coverage error, or undercoverage, and no random draw can repair it, because a random sample of a bad list is a random sample of the wrong population. The voter register in the vaccination example misses adults who are not registered; a hospital payroll misses agency and bank staff; a landline directory misses mobile-only households. The second gate is selection: even within a good frame, some members may have a higher chance of being picked, as when a nurse replaces a missed appointment with the next convenient patient. The third gate is response: a low return rate is not itself a bias, it is only a warning light. What makes non-response bias is whether the tendency to answer is related to the quantity being measured. A 90 % response rate can be badly biased if the silent 10 % are exactly the people with the outcome, and a 50 % rate can be almost unbiased if answering is unrelated to it. Two tests are standard. Compare the responders with what is already known about the population on variables you hold for everyone, such as age, sex or ward. Then compare early responders with late responders, because late responders usually resemble the people who never answered. Several named biases live inside these gates: self-selection or volunteer bias, when people put themselves forward; social-desirability bias, when respondents give the answer they think is expected; recall bias, when memory is better for one group than another; and interviewer bias, when the person asking influences the answer. The arithmetic that makes non-response vivid is the worst-case range. If 700 of 1,000 invited patients return a questionnaire and 30 % of responders report insomnia, the truth lies between 0.70 × 0.30 = 0.21 = 21 %, if no non-responder had insomnia, and 0.21 + 0.30 = 0.51 = 51 %, if all of them did. The survey cannot say more than that, however tidy the 30 % looks.
Reading a survey like a nurse: the checklist, and reliability against validity: The seven decisions become a practical routine once you know what each one protects against. Start with the target population: write it as a rule of person, place and time, then ask whether the sampling frame actually lists every member of that group. Next the sampling method: a random draw from a complete frame removes selection bias, while a convenience draw does not, and the report should say which was used. Sample size sets the random error only. For a proportion the margin of error near p = 0.5 is roughly 1 ÷ √n, so n = 400 gives about 1 ÷ 20 = 0.05 = 5 % and n = 1,000 gives about 3.2 %, and no sample size touches a bias of even two percentage points. Questionnaire design is where most avoidable error enters. A leading question suggests the answer it wants; a double-barrelled question joins two issues so that a single reply cannot be interpreted; an ambiguous reference period, such as “recently”, means different months to different people; response options that are not exhaustive or that overlap force people into the wrong box; and question order can change what people say. The cure is cheap: pilot the questionnaire on a few patients, ask them what they thought each question meant, and rewrite it. The sixth decision, sources of bias, is the honest statement of what could not be repaired. The seventh, reliability of measurements, needs two separate ideas. Reliability is repeatability: would the same instrument or the same observer give the same value again? It is tested by repeating the measurement on the same people after a short interval, test-retest, or by having two observers measure the same patient, inter-rater agreement. Validity is a different question: does the measurement capture the thing you care about at all? A pain scale can be perfectly reliable, giving the same score every time, and still be invalid for measuring function. This distinction has a direct nursing use. Two nurses who score the same patient's pain three points apart have a reliability problem, solved by training and a written standard; a scale that consistently reads 0.8 kg too high has a validity and calibration problem, solved by servicing the scale. Neither is solved by enlarging the sample, and both must be reported alongside the percentage.

Explore the diagram

Look at the whole drawing first. Then select one numbered part and connect it to the idea you just read.

Random error versus bias in a sample

Two bullseye targets drawn side by side. The left target shows random error: the plotted observations spread around the true centre in every direction, so adding more observations tightens the cloud. The right target shows bias: the observations are tightly grouped but the whole group sits away from the centre, so adding more observations keeps the same tilt. Below, the three gates through which bias enters a sample are named with clinical examples, and the seven survey design decisions are listed with the arithmetic of a response rate of 75 % and its worst-case range.

New to diagrams? Read this first

Look at the whole drawing once without clicking anything. Ask yourself only: what kind of picture is this (a comparison, a route, a sequence, a table)?
Press Play, or use the arrows, to move through the numbered parts in order. The drawing stays still; only the highlight moves.
Click a number, or its name in the list below, to read what that part is and why it matters. Use “Hide labels” when you want to test your memory.

More controls

100%

01 / 4Random error: the cloud tightens

If the drawing is wider than this panel, scroll it sideways. Labels stay readable; the numbered explanations below show the same ideas.

Random error: the cloud tightens

The green dots scatter around the true centre in every direction, which is what random error looks like. Because the misses point both ways, they partly cancel: as observations are added, the average settles closer to the centre and the cloud becomes denser. The precision of an estimate therefore improves with sample size. The standard error of a mean, s ÷ √n, is the arithmetic of this panel, and quadrupling the sample halves it.

Schematic bullseye targets: the dots are illustrative observations, not measurements from a real study. Their positions show the kind of error, not its exact size. The response-rate box uses a questionnaire sent to 800 people with 600 returns and 45 % of responders reporting the outcome.

Key points to remember

Sampling bias is a fault in the method that systematically produces a sample unlike the target population, so the result is pushed in one direction.
Random error shrinks as the sample grows; bias does not, because every extra person is chosen or questioned in the same wrong way.
Selection bias is in who can be chosen, non-response bias is in who answers, and convenience bias is in who is easiest to reach.
A survey is a systematic method of collecting information; a poll is a survey designed to measure opinions, preferences or characteristics.
Seven decisions keep a survey honest: target population, sampling method, sample size, questionnaire design, response rate, sources of bias and reliability of measurements.
Random selection protects only the gate of selection; a leading question, a low response rate or a faulty instrument still distorts the result.

Go deeper

Worked example

Reporting a staff burnout survey honestly. A 1,200-bed hospital wants to know what proportion of its 2,400 nurses have symptoms of burnout. The occupational-health service emails a questionnaire to all 2,400 nurses. After four weeks, 960 replies have arrived, and 384 of those responders score above the burnout threshold. The service must report the result without overstating what it knows.

Name the target population and the group actually measured. The target population is all 2,400 nurses on the payroll; the sample obtained is the 960 nurses who replied.
Compute the response rate: answers ÷ people asked = 960 ÷ 2,400 = 0.40 = 40 %.
Compute the proportion among responders: 384 ÷ 960 = 0.40 = 40 %. This 40 % describes the 960 responders exactly and the whole nursing body only if replying is unrelated to burnout.
Reason about the direction. The most exhausted nurses may be the least likely to open a long email, which pushes the estimate down; nurses with a grievance may be the most likely to reply, which pushes it up. Email-only contact also misses nurses on long leave. The direction is therefore uncertain, and that uncertainty must be stated rather than hidden.
Bound the answer rather than defend a single number. If none of the 1,440 non-responders had burnout, the hospital proportion is 0.40 × 0.40 = 0.16 = 16 %. If all of them did, it is 0.16 + (1 − 0.40) = 0.16 + 0.60 = 0.76 = 76 %. The data cannot separate 16 % from 76 %.

The honest report gives the target population, the method, the response rate of 40 %, the responder proportion of 40 %, and the range from 16 % to 76 %. The number to avoid is the bare sentence “40 % of our nurses are burnt out”, because with a 40 % response rate that sentence claims far more than the survey can support.

Tutorial 1 — finding the bias in a recruitment method and fixing it

A researcher needs the mean systolic blood pressure of the 2,400 adults registered at a health centre. She puts a poster in the waiting room and asks people to sign up at the reception desk between 9 a.m. and 4 p.m. on weekdays. In three weeks, 180 adults volunteer and are measured. Before she reports a mean, she must find out what her method has done to the sample.

Rule used: A sample is unbiased for a quantity when every member of the target population has the same chance of entering it. Check three gates in order: who can enter at all (the sampling frame), who is actually chosen, and who stays in once chosen. The sample may be generalised to the target population only when all three gates are open to that whole population.

Write down the target population and the group that could actually enter this sample.
Reveal this step
The target population is the 2,400 registered adults. The accessible group is much smaller: only adults who come to the centre in person, on a weekday, between 9 a.m. and 4 p.m., and who are willing to sign a form at reception. Working adults, the housebound and anyone attending only in the evening cannot enter at all.
Which named bias does a poster plus a reception desk during office hours produce, and in which direction is the mean blood pressure likely to move?
Reveal the answer
This is convenience sampling bias, with a strong element of selection bias: participants are chosen because they are easy to reach. Adults who attend a health centre during working hours are more likely to be unwell, older, retired or already worried about their blood pressure, so the mean is likely to be pulled above the true value for all 2,400 adults. The safe statement is that the direction is not random; the likely direction is upward, and that reasoning should be written down.
List who the method excludes, and name the gate each exclusion belongs to.
Reveal this step
The centre's own registration list is not used, so the first gate is only partly relevant; the method bypasses the frame altogether. The second gate excludes everyone not present in the building during office hours. The third gate excludes people who see the poster but do not volunteer, and volunteers are not the same people as non-volunteers.
Propose one concrete fix for the frame, one for selection and one for non-response.
Reveal the answer
For the frame: obtain the complete register of all 2,400 adults, including addresses and telephone numbers. For selection: draw a simple random sample from that register with a random-number table or a computer, so choice does not depend on being present. For non-response: contact the people drawn with appointments at different times, including evenings and weekends, make two follow-up attempts and record the final response rate.
Write the one sentence the researcher should include with her mean.
Reveal this step
The sentence should state the size of the sample, who was eligible, how they were chosen, the response rate, and the direction of the remaining bias. For example: “Among 180 volunteers recruited from the waiting room, the mean was 138 mmHg; because participants volunteered during office hours, this value probably overestimates the mean of all 2,400 registered adults.”

Answer: The poster method produces a convenience sample tilted upwards; the honest fix is a random draw from the full register, appointments at varied times and two follow-up attempts.

Keep this habit: Do not read a recruitment method and ask only how many people took part. Ask who could enter, who was actually chosen and who stayed: each gate can fail separately, and only the gate that failed tells you which repair to make.

Tutorial 2 — sampling bias or non-response bias?

A city health office wants the proportion of adults vaccinated against influenza. It draws a simple random sample of 1,000 names from the voter register, mails each a short questionnaire, and receives 480 replies. Among the responders, 62 % say they were vaccinated. The office must decide which faults belong to the drawing stage and which belong to the answering stage.

Rule used: Sampling bias is present when the method of selection keeps some members of the target population out of the sample. Non-response bias is present when, among people who were correctly selected, those who answer differ systematically from those who do not. Identify the stage first: drawing, or answering. When a survey gives only the proportion among responders, bound its worst case: lower bound = response rate × proportion among responders; upper bound = that value + (1 − response rate), where (1 − response rate) is the silent share, assumed in turn to have none and then all of the outcome.

State the target population, the sampling frame and the response rate.
Reveal this step
The target population is all adults in the city. The frame is the voter register, which contains only registered adults. The response rate is answers ÷ asked = 480 ÷ 1,000 = 0.48 = 48 %, so slightly more than half of those selected stayed silent.
The names were drawn at random from the register. Does that remove every form of selection bias?
Reveal the answer
No. Random selection protects only within the list that was used. The register itself is an incomplete frame: adults who are not registered, who include many young and mobile residents, have a zero chance of selection. That is coverage error, a form of selection bias, and it remains however random the draw inside the list.
Suppose every one of the 1,000 selected adults had returned the questionnaire. Which bias would disappear, and which one would remain?
Reveal the answer
Non-response bias would disappear, because everybody selected would have answered. The frame problem would remain: the study would still describe only registered adults, so it could not speak for the unregistered part of the city population. A perfect response rate repairs one gate and leaves the other untouched.
Compute the worst-case range for vaccination coverage from the response rate of 48 % and the 62 % reported by responders.
Reveal this step
If none of the non-responders was vaccinated, the overall proportion is 0.48 × 0.62 = 0.2976 ≈ 29.8 %. If all of them were, it is 0.2976 + (1 − 0.48) = 0.2976 + 0.52 = 0.8176 ≈ 81.8 %. Without more information about the silent half, the survey cannot separate roughly 30 % from roughly 82 %.
State the two fixes, one for each bias.
Reveal this step
For the frame, use a list that covers the whole adult population, such as a population register or a random-digit mobile telephone frame, and say which populations remain outside it. For non-response, offer several ways to answer, send two reminders, and compare early with late responders, since late responders often resemble the people who never answered.

Answer: The survey carries selection bias from an incomplete voter frame and non-response bias from a 48 % response rate; the responder value of 62 % bounds coverage only between about 30 % and 82 %.

Keep this habit: Ask two separate questions of every survey: could everyone in the target population be drawn, and did the people who were drawn actually answer? Random selection answers the first question only within the list you used, and it says nothing at all about the second.

Now you try

Attempt each one on paper first. Write the formula, substitute the numbers, then check the unit of your answer.

A physiotherapy department wants the mean pain score of all 900 patients it treated in a year. A student stands in the gym on Tuesday morning and asks the 60 patients who happen to be there. Name the bias, state the likely direction, and give one concrete fix.
Hint
Write down who could possibly be in that room on a Tuesday morning, then list everyone who is missing.
Show the worked answer
This is convenience sampling bias with a selection element. Patients present on a Tuesday morning are those with an appointment and enough mobility to attend; patients too unwell to travel, those seen at home, those discharged and those with evening appointments have no chance of entering. Pain scores are likely pulled towards the levels of people well enough to attend, so the direction is probably downward. The fix is to obtain the list of all 900 treated patients and draw a simple random sample, contacting non-attenders by telephone or post.
A postal questionnaire is sent to 800 randomly selected patients. 600 return it, and 270 of those responders (45 %) report a side effect. Give the response rate and the worst-case range for the true proportion among all 800.
Hint
Response rate is returned ÷ sent. Then calculate the overall proportion twice: once with every non-responder saying no, once with every non-responder saying yes.
Show the worked answer
Response rate = 600 ÷ 800 = 0.75 = 75 %. The 45 % describes only the responders. If none of the 200 non-responders had the side effect, the overall proportion is 0.75 × 0.45 = 0.3375 ≈ 33.8 %. If all of them did, it is 0.3375 + (1 − 0.75) = 0.3375 + 0.25 = 0.5875 ≈ 58.8 %. The survey therefore bounds the truth between about 34 % and 59 %, and a larger sample of the same kind would narrow only the random part, not that range.
A study draws a true simple random sample of 3,000 adults, asks “Do you agree that the government wastes money on hospitals and that waiting times are too long?”, and measures waiting time with a clock that runs 5 minutes fast. Explain what random sampling fixes here and what it does not.
Hint
Separate who was chosen from how they were asked and how the time was measured.
Show the worked answer
Random selection reduces selection bias within the sampling list: every adult on the list had an equal chance, so the people chosen are not tilted by convenience. It does nothing after that. The question is both leading and double-barrelled, inviting agreement and joining two issues, so the answers carry a measurement error that no sample size removes. The fast clock adds a constant 5-minute error to every reading, so the mean waiting time is about 5 minutes too high whether the sample holds 3,000 or 30,000 adults. Random sampling makes errors behave randomly only when the question and the instrument are sound.

Common misconception

Beginners often believe that a bigger sample always makes a study better, so a survey of 10,000 people must be more trustworthy than one of 500. Size controls only the random part of the error. A biased method pushes every one of those 10,000 people in the same direction, so the large survey converges confidently on the wrong value and its confidence interval becomes narrow around that wrong value. The hospital burnout survey is the everyday version: 40 % of responders reported burnout, but with a 40 % response rate the true figure could lie anywhere from about 16 % to about 76 %. Precision is not accuracy, and a large biased sample is the most convincing kind of mistake.

Check your reasoning

A colleague says: “Our survey had a 30 % response rate, so we should simply increase the sample from 500 to 5,000 people to make the result reliable.” Explain what that change would and would not achieve.

Reveal an answer

Enlarging the sample multiplies the number of people invited, so it shrinks the random error and makes the estimate more precise. It does nothing about the 70 % who would still not answer, and nothing about who those people are. If non-response is related to the quantity measured — the most exhausted nurses not replying, for example — the same tilt is reproduced in a larger sample, and the interval around the wrong value becomes narrower and more convincing. The change that actually helps is not size but follow-up: reminders, several ways of answering, and a comparison of late responders with the rest, so the silent group can be partly understood rather than multiplied.

Where this comes from

BMS 1110 Applied Mathematics and Biostatistics — supplied lecture notes, Unit 6 biostatistics block · Lines 1083–1105: sampling bias defined as the method of selecting a sample systematically producing a sample that does not adequately represent the target population, with misleading conclusions (1085–1086); selection bias, non-response bias and convenience sampling bias listed (1088–1090); the fitness-centre obesity example (1092–1093); a survey as a systematic method of collecting information from individuals and a poll as a survey designed to measure opinions, preferences or characteristics (1095–1096); the seven design decisions — target population, sampling method, sample size, questionnaire design, response rate, possible sources of bias and reliability of measurements (1098–1104); and the warning that a random sample does not automatically eliminate every source of error, since poorly designed questions, non-response and measurement problems can still affect results (1105).
Standard survey-methodology practice for the deeper explanations · the sampling frame and coverage error or undercoverage; the three gates of frame, selection and response; the worst-case non-response bounds response rate × responder proportion (lower) and that value + (1 − response rate) (upper), worked as 21 % to 51 % at a 70 % response rate and 30 % among responders, and 29.8 % to 81.8 % at a 48 % response rate and 62 % among responders; the decomposition of error as mean squared error = variance of the estimate + bias², worked as 3² + 1² = 10 falling only to 9 + 0.25 = 9.25 when the sample is quadrupled; the names self-selection or volunteer bias, social-desirability bias, recall bias and interviewer bias; the margin of error for a proportion near p = 0.5 as roughly 1 ÷ √n, giving about 5 % at n = 400 and 3.2 % at n = 1,000; and the distinction between reliability (test-retest, inter-rater agreement) and validity in measurement.

Lesson progress is saved on this device for the current learner. Practice answers stay separate and are not stored.

About this course and its sources

Bilingual lessons organised from the supplied BMS 1110 Applied Mathematics and Medical Biostatistics lecture notes (Mr. Ebangan Prosper Achuanoh). Diagrams are original educational schematics; sections that go beyond the supplied notes say so in their sources. Mathematics, clinical and French-language review are pending.