Yes, a published taxonomy exists, and it is a strong basis for this work. The Force Concept Inventory (FCI), published by Hestenes, Wells and Swackhamer in The Physics Teacher in 1992, comes with a taxonomy of 28 commonsense misconceptions in six major categories, each linked to the inventory items whose wrong answers suggest it. If you build red-team prompts from those categories, each test case can be traced to a category in a published, peer-reviewed taxonomy. Whether coverage is “enough” then becomes a scoping judgement you defend with a mapping to your syllabus, not a test-case count you have to reach.
TechnoLynx can build misconception-driven evaluation harnesses for RAG and LLM systems within its Generative & Agentic AI R&D practice. The work covers four things: deriving the adversarial prompt suite from the taxonomy, mapping its categories to the topics your course or application must cover, running the evaluation harness against the system, and documenting which categories are covered so the coverage claim holds up in review. The crosswalk from taxonomy categories to your topic list defines the scope of the evaluation harness.
Is there a published taxonomy of physics misconceptions I can design test prompts from?
There are several layers of it, and they build on each other. Halloun and Hestenes published a taxonomy of common-sense concepts that conflict with Newtonian theory in the American Journal of Physics in 1985. Their classification follows Newtonian mechanics itself: misconceptions about principles of motion (alternatives to Newton’s laws) and misconceptions about influences on motion (alternatives to specific force laws).
The 1992 FCI covered those concepts more systematically and made results easier to interpret. Its Table I splits the Newtonian force concept into six conceptual dimensions: kinematics, the first law, the second law, the third law, the superposition principle and kinds of force. Its Table II groups the misconceptions into six categories that match those dimensions as closely as possible. Hestenes and Jackson later refined the taxonomy. Hestenes and Halloun describe the tables as “crucial to interpreting results of the FCI”.
To write prompts, the coded form is the most useful. Escalante’s version lists 30 codes in named sections, for example K2 “velocity-acceleration undiscriminated”, I5 “circular impetus”, AF2 “motion implies active force”, AR1 “greater mass implies greater force”, CI1 “largest force determines motion” and G3 “heavier objects fall faster”. That paper attributes 30 misconceptions to Hestenes et al., while the 1992 paper says 28. Pick one version, cite it by table, and keep your coverage claims consistent with it.
Why these categories are more than one author’s opinion
The distractors come from students’ own answers to open-ended versions of the questions. About half the FCI questions come from the earlier Mechanics Diagnostic Test, whose questions were built from students’ open-ended answers, reviewed by experts and refined through interviews. The multiple-choice versions used the most common written responses showing non-Newtonian beliefs as their wrong options.
The FCI holds Gold Star validation, the highest level of research validation. That rating requires research into student thinking, expert review, appropriate statistics, work at several institutions by several groups, and peer-reviewed publication. PhysPort also counts more than 50 published studies using the FCI, at over 70 institutions, with data on over 35,000 students. The original authors also say that “errors” on the inventory tell you more than correct answers do. That is the property a red-team needs.
Later network-analysis work points the same way. A network analysis of wrong-answer patterns, called Modified Module Analysis, found communities of incorrect answers that in most cases mapped back onto the misconceptions the FCI’s authors used to build it.
Using the Force Concept Inventory to probe a language model
The current FCI (v95) has 30 questions. Each item has five choices: one correct answer and four distractors. The items cover kinematics, projectile motion, free fall, circular motion and Newton’s laws, roughly the first third of a college introductory semester. Because each distractor is tied to one or more coded misconceptions, a model’s wrong answer is a diagnosis, not just a lost point.
The evidence that language models repeat these specific errors is mixed. Wheeler and Scherr (PERC Proceedings) found ChatGPT averaged 51% on the FCI, did worst on kinematics and Newton’s second law, and gave explanations that treated vectors as scalars, confused momentum with force and misread forces in outer space. One study tested 3,202 multiple-choice questions with real student response distributions. When models answered wrongly, they picked the students’ most common wrong answer about 51% of the time at 0.5B parameters and up to 59% for larger models.
The stronger models complicate the picture. West (2023) reported that GPT-3.5 scored 15 of 30 on its first attempt and GPT-4 scored 28 of 30. GPT-4 also seemed unable to reproduce novice reasoning even when asked to pretend it did not know Newton’s laws. When ChatGPT 3.5 was asked to answer “as a novice”, its answers matched the most common student pretest answers on 11 of 23 items. On West’s weighted measure it scored 63 against a random-guess floor of 58.
So a high FCI score from a frontier model does not show that the misconceptions are absent from what your system produces. It shows the model handles the canonical wording. The FCI’s designers probed each concept from several angles and with small variations, and West notes that GPT-3.5’s swings between expertise and nonsense show why that matters. In a red-team suite, each taxonomy category should therefore seed several prompts rather than one. Use the original item adapted to text, a paraphrase with a new everyday context, an open-ended “explain why” version, and a turn where the user states the misconception as fact and the system has to correct it. That is another reason to score the paraphrased variants separately from the originals.
The student score thresholds do not carry over either. Hestenes and Halloun proposed 60% as the entry point to Newtonian thinking and 85% as mastery for students. Those numbers mark stages in how students learn Newtonian mechanics. They are not a pass mark for a deployed system, where one confidently wrong explanation in a high-traffic topic can matter more than the total score.
There is precedent for this approach. MisconceptionBench (EMNLP 2026) evaluates LLMs on 761 K-12 science items with 1,474 expert-annotated distractors and response distributions from more than 150,000 students, and concludes that misconception-centred reasoning needs evaluation beyond answer accuracy. Siddiq et al. applied concept-inventory methods to software security and derived a 51-item taxonomy of recurring LLM misconceptions. They also warn that such a taxonomy is a snapshot of current models, not a stable set.
Which concept inventories map onto my syllabus topics?
The physics and astronomy education communities have produced more than 60 research-based assessment instruments. No single mechanics instrument covers all the content of an introductory mechanics course. Each has a narrower range so it can probe its sub-topic thoroughly. You will usually need several instruments.
| Syllabus topic group | Research-based inventories | What to note |
|---|---|---|
| Kinematics and forces | FCI, FMCE, TUG-K, MBT, FVA, IBCM | Six instruments; the FCI is one of the earliest |
| Motion graphs | TUG-K | Position, velocity and acceleration versus time graphs |
| Energy | EMCS, ECA | Introductory university level |
| Rotation and rolling | RRMCS, RKI | Torque, moment of inertia, rolling; RKI in three parts |
| Introductory electricity and magnetism | BEMA, CSEM, DEEM, MCS | DEEM has 66 questions against 31 and 32 for CSEM and BEMA |
| DC circuits | DIRECT | Resistive DC circuit concepts |
| Quantum mechanics | QPCS, QMCS, QMCI, QMCA | In the QMCI, each answer choice is tied to a specific alternative conception |
Even within mechanics, the FCI and FMCE split the territory differently. The FCI covers more topics, while the Force and Motion Conceptual Evaluation (FMCE) asks more questions per topic. The FCI also adds two-dimensional motion, impulsive forces, vector sums and identification of forces. Thornton et al. found that 22 of the 30 FCI items fall outside the FMCE’s coverage. The current FMCE (v99) has 47 questions, and factor analysis groups them around Newton’s first and second laws, Newton’s third law, and the velocity concept. A mechanics syllabus that includes projectiles and circular motion is better matched by the FCI than by the FMCE, which omits two-dimensional and circular motion. A syllabus that only needs deep coverage of one-dimensional dynamics may be better served by the FMCE.
Building the crosswalk, step by step
- List the syllabus or application topics at the level of detail your approval claim will be made at. “Newton’s third law in collisions” is checkable; “mechanics” is not.
- For each topic, record the instrument, its version and the table its categories come from. A topic with no instrument is a gap, not an omission to hide.
- Map taxonomy categories to topics, and expect overlaps. Network analysis of FCI answers found that items in one wrong-answer community often belong to several misconception categories.
- Write down each mapping judgement with its reason. This record is what you show a reviewer.
- Generate several prompt variants per mapped category (adapted item, paraphrase, explanation, user assertion), and keep the category code on every prompt.
- Score each response by category: did the system reproduce the misconception, correct it, or avoid it? Accuracy alone does not tell you which.
Our wider discussion of evaluation harnesses that decide red-teaming and RAG questions covers how to turn per-category results into a release decision.
How many misconception categories are enough to claim coverage?
No fixed number of misconception categories makes coverage sufficient, and we would be wary of anyone who quotes one. In Escalante’s study, the misconceptions that showed up differed between courses: 17 of the 30 coded misconceptions were dominant wrong answers in one course and 15 in another.
A defensible coverage claim therefore sounds like this: every in-scope topic is represented by at least one mapped category, each category has several prompt variants, and out-of-scope topics are listed by name with the reason. Five hundred prompts on Newton’s third law and none on kinematics is a large suite with poor coverage. Fifty prompts spread across every mapped category in your syllabus is a smaller suite with a much better claim. Hestenes and Halloun advise using the whole FCI for course evaluation because the total score measures coherence across all dimensions. The equivalent discipline here is not to pick favourable categories within an instrument you have declared in scope.
Three limits bound what any coverage statement can say. The FCI is a survey instrument: some issues are tested by only a few questions in one or two contexts, so it cannot pin down which beliefs a single student holds or when. The taxonomy is more detailed than the way students actually apply misconceptions in their answers. The FMCE study also found conceptions outside the FCI taxonomy, which suggests students hold a richer set than the FCI measures. Wheeler and Scherr state plainly that the FCI does not represent the full range of introductory physics topics. Saul’s review adds that the FCI should be one part of a broader assessment, not the only one.
Why a citable taxonomy makes stronger approval evidence than ad-hoc cases
A suite of ad-hoc cases has a hidden problem: a reviewer cannot tell what was left out, because the gaps are wherever the test author’s own understanding had gaps. A literature-derived taxonomy turns that into a question the reviewer can check. The categories exist independently of your team, and the crosswalk shows which ones you tested and which you excluded. Each excluded category is visible and has to be argued for.
That structure matches what NIST asks of red-teaming. NIST AI 600-1, the Generative AI Profile of the AI Risk Management Framework, tells organisations to define the use cases, contexts of use and negative impacts where red-teaming would help most. It also asks for risk measurement in context of use with standardised measurement protocols, and for adversarial role-play or red-teaming to find unexpected failure modes. A confidently wrong physics explanation fits the profile’s definition of confabulation: erroneous content presented with confidence. The taxonomy mapping supplies the “context of use” and “standardised protocol” parts in a form a committee can read.
TechnoLynx treats approval-grade evidence as an engineering output in its own right: an artefact a buyer, auditor or review committee can read and sign against. This evidence is not certification or regulatory sign-off. It produces the artefacts a review needs. The physics judgement in the crosswalk also benefits from a subject expert, such as an instructor who knows the course, signing off the mapping. A course team with that expertise and an existing evaluation pipeline can reasonably do this work in-house.
One boundary is worth stating plainly. Misconception coverage is a different problem from security red-teaming. NIST lists prompt injection, data poisoning and model extraction as separate targets for red-teaming, and suites built for those risks answer a different question.
Which physics concepts is a model most likely to get wrong?
The student literature gives a starting list. Hestenes, Wells and Swackhamer name the impetus concept of motion and the dominance principle as the hardest misconceptions, usually the last to be overcome. In Escalante’s data, impetus was the most dominant section. On FCI item 4, the truck-and-car collision, nearly 75% of West’s 2018 pretest students thought the truck exerts the greater force. Halloun and Hestenes call the belief that heavier objects fall faster so common that it deserves careful attention in class. They also identify failure to tell kinematic quantities apart as the most common and critical kinematics problem.
The model-specific evidence overlaps with that list in some places and not in others. ChatGPT did worst on kinematics and the second law, and showed velocity-acceleration confusion and scalar addition of velocities. Its explanations also confused momentum with force and misread forces in outer space. Results differ between ChatGPT versions, so we would weight the suite towards these categories at first and re-derive the weighting from each new model’s per-category results, not carry it forward.
The weak point of this method is the crosswalk, where a person decides that a syllabus topic is “covered” by a category that only partly fits it. That is where review effort should go. Outside mechanics, which has the most inventories, more of the coverage claim rests on that judgement, and the evidence pack should say so openly.
