Beyond 1+2=3

Auto Scoring for Open-Ended Math Questions

Part 3 of a 5 part series exploring the benefits of technology-enhanced items in education & assessment. Brought to you by Learnosity and A Pass Education Group.

Technology-enhanced items are continuing to grow in popularity.

Modern-day assessments now have the appropriate tools and features to allow for much more challenging and innovative questions than ever before. Learner engagement levels are significantly increased, and assessors have the ability to test a variety of skill sets that were previously untestable. One of the most beneficial features offered to assessors is the impressive auto-scoring capabilities of technology-enhanced items.

This paper explores the capabilities of the Learnosity math & chemistry scoring engine used in technology-enhanced items, demonstrating the many advantages of this feature from both an educator and student perspective. From basic formulas to the more complex mathematical expressions, the powerful auto-scoring math engine enables advanced, rule-based auto-grading.

Math Scoring Basics

The smart math scoring feature of the Learnosity math engine permits question creators to ask open-ended math questions. The key element of this feature is that these answers are auto-scorable; the author of the question can define a set of parameters for the correct answer and, as long as the student’s response fits these parameters, the question will be scored as correct. This means that students can enter their answer in a wide variety of forms and syntaxes and will not be penalized for using different variable ordering, bracketing, or spaces between elements. There is both an onscreen keyboard, which allows the input of symbols not readily available on standard QWERTY keyboards, as well as a handwriting feature, which enables students to utilize a touchscreen interface. Calculators and measuring tools such as rulers and protractors are also available.

Protractor for Math Scoring TEIScientific Calculator in Technology Enhanced Items

The scoring feature can read numbers, letters, symbols, expressions, and equations.

The author of the question can define how strictly students must adhere to a specific form. For example, the question can be set so that “15” scores as correct, but an unsimplified form like  10 + 15 will score as incorrect. Similarly, if a question states that an answer must be written in expanded standard form, an author can set  x^2 + 4x + 4 to score as correct, but  left(x+2right)left(x+2right) (the factored form) or x^{2}+4 +4x (an equivalent expression not in standard form) to score as incorrect. Conversely, the author can also allow flexibility in answering a question, such as allowing either 4x-1 or  -1 +4x to validate correctly. There are a variety of other specific validation features that will be discussed in-depth later in this brief.

Advantages of Smart Math Scoring

There are several important advantages of this functionality. Most significantly, it allows for the assessment of higher levels of mathematical skills. Whenever multiple-choice or another structured response format is involved, there is always the possibility that students can simply guess the correct answer. Furthermore, although authors seek to avoid this pitfall, sometimes answer choices can give students hints that guide them to the correct answer without having to demonstrate that the skill being tested has been mastered. For example, if a student is asked to factor a quadratic and all of the answer choices contain radical expressions, this gives students a clue that they will need to use the quadratic formula to correctly factor the equation. Additionally, when students perform computations and then do not find their answer among the choices given, they know they made an error and can reevaluate the steps they took to reach their answer. Use of open-ended math questions effectively addresses each of these concerns.

A second advantage is higher author efficiency. When crafting questions in a structured-response format, such as multiple choice or drag-and-drop, the question author must craft distractors that do not provide clues that allow the student to easily guess the correct answer. This often involves creating many distractors, such as numerous possible factors, both positive and negative, between 0 and 10. This process can be time-consuming. Additionally, the different validation features, such as allowing equivalent forms of an expression to validate as correct, reduce authoring time by allowing the question creator to simply adjust a setting, rather than having to manually enter all of the possible equivalent forms.

These features not only save time, they also reduce the risk of human error on the part of the author.

Smart Math Scoring in Action

To start with a basic example, students can be asked to add or subtract integers.

Using the “EquivLiteral” setting, only the correct answer, entered exactly as it was by the author, will score correctly. If the question asks for the value of 27-9, only 18 will validate.

Similarly, students may be asked to add two fractions, such as left(frac{1}{3}right)+left(frac{5}{12}right). In this case, if the author uses the “EquivValue” scoring method, students will be able to enter either left(frac{9}{12}right), the unsimplified form, or left(frac{3}{4}right), the simplified form or, indeed, 0.75, the decimal form. If the author only wants to accept the simplified fraction, he or she may use the “EquivLiteral” setting.

At a higher level, students may be asked to create equivalent equations, such as manipulating a quadratic equation from standard form to vertex form. In this case, the author would use “EquivLiteral” because vertex form requires the answer to be in a specific order. On the other hand, if students are asked for the product of x+2 and x-5, the author can use the “EquivSymbolic” setting, which will accept all equivalent forms of the expression product of (left(x+2right)left(x-5right). In order to make sure students cannot enter (x+2)[math](x-5)[/math] as an answer (which “EquivSymbolic” would validate as correct), the author could use an additional setting, “IsExpanded,” which will only accept answers that are in the expanded, rather than factored, form. Similarly, there is an “IsFactorized” setting, which operates in the opposite way, only accepting factored, rather than expanded, forms. There is a similar “IsSimplified” setting, which would accept x^2-3x-10, but not x^2-5x+2x-10.

The scoring feature can handle more complex expressions and symbols, such as radical expressions and pi (pi). Students can enter answers such as 3pmsqrt{7} or 72pi. Question creators can give students a complex equation, such as F=Gfrac{m_1m_2}{d^2}, Newton’s universal law of gravitation, and ask students to write the equation in terms of left(G=Ffrac{d^2}{m_1m_2}right). Students can also enter logarithm answers, such as log_{10}{}x.

Question creators can provide students with a variety of keyboards to allow them to enter nearly any symbol or mathematical operation needed, including Greek letters, absolute values, trigonometric functions, integrals, and matrices. Authors may also choose to create custom keyboards.

Using the alternate response feature, authors can ask more open-ended questions. For example, if an equation has three factors, left(x+3right), left(x-2right), and left(x+1right), the author can ask a question such as “What is one factor of x^3+2x^2-5x-6?” and allow any of the above factors to validate as correct. Question creators also have the flexibility to ask for any equivalent expression, and allow any equivalent expression to validate except the original expression given in the question stem by using the “inverse result” setting. There are numerous other options, including allowing an answer such as 1245 to validate correctly with or without the comma as the thousands separator, or allowing (or disallowing) the use of a decimal or a fraction (such as 0.5 or frac{1}{2}) as a correct answer, or even setting the number of decimal places required for a correct answer.

Sometimes, question creators will need to use a combination of settings, such as the “EquivSymbolic” and “inverse result” to allow any equivalent expressions except for the one given, or “EquivLiteral” and “ignore order” to allow certain terms to be entered, but in any order (such as 2x+1 or 1=2x). Learnosity has the flexibility to accommodate these combinations as needed, allowing for nearly endless possibilities. This level of customization makes the math-scoring feature a powerful tool that can be used in a variety of situations to suit the needs of the assessor.

Smart math scoring can be used in many different question scenarios. Rule-based, math grading can be used in multiple different math question types. Depending on the grade level of the student and the pedagogical intention of the question it may be appropriate to format the question differently. For example the student could be asked to fill in the blanks or label an image rather than being asked a completely open-ended question

Additional Features

Handwriting Recognition Technology

All Math Formula questions can be enabled with handwriting recognition functionality.

This makes it significantly easier for students using touchscreen devices such as iPad and Android tablets to enter complex Math Formulas.

Handwriting Recognition for Math TEIStudents “write” the required symbols, and the handwriting recognition technology will translate them into machine-readable digital information. All student responses are then scored using the smart math scoring detailed above.

This provides students with a more natural and accessible math learning experience while also giving them all the cognitive benefits of handwriting.


The smart math-scoring feature offers several significant advantages to students and educators. One of the most powerful advantages of the feature is that it allows for the assessment of higher levels of mathematical skills through the use of open-ended questions for complex expressions and symbols that are frequently used in advanced math and chemistry. It also enables higher author efficiency through the use of various validation features that save time and simultaneously reduce the risk of human error on the part of the author.

Interested in learning more about technology-enhanced items in assessment? Our next angle takes an in-depth look at the ways in which question authors can use hints and worked solutions to provide on-demand feedback and assistance to students. The feature is beneficial for both students and assessors. It can assist students with questions they may not otherwise have been able to answer, and educators can see which students viewed which hints, which in turn provides them with insight on where they should target their instruction.
To learn more, look out for the next release of our technology-enhanced items series: “A Helping Hand: The Use of Hints & Worked Solutions in Assessment.”


Download Button for Math Scoring of Technology Enhanced Items Article

SAVE THE DATE: Interactive webinar, November 19th 1pm (EST)

Don’t Just Check the Box – Technology Enhanced Items in Assessment

Explore the overall benefits of using technology-enhanced items in assessment, and get interactive demo of the authoring experience for some of the more advanced technology-enhanced solutions such as spoken response, rule-based grading for math and chemistry, creating worked solutions and hints for students and adaptive testing.

Button to Register for Technology Enhanced Items Webinar


 About Learnosity:

Learnosity Assessment SoftwarePartnering with companies across a range of sectors—including K-12, Higher Education, and Corporate Education—Learnosity provides the technology framework for authoring, assessment delivery, and reporting for many of the world’s best assessment solutions. Leveraging the Learnosity offerings enables clients to enhance any digital product, new or existing. It also reduces the need to reinvent the wheel, significantly increases speed to market, and decreases the overall cost of ownership.

Learnosity offers one of the widest ranges of technology-enhanced items on the market, with over 55 distinct question types currently available, ranging from the more basic fill-in-the-blank question types to spoken-response capture, Cartesian graphing, handwriting-recognition technology as well as advanced Math and Chemistry formulas. The authoring experience is designed to allow any user—professional content author or the more casual teacher author—to create advanced technology-enhanced items in minutes. It’s as simple as using any word processor. A wide range of implementation and storage solutions are also available. Clients can choose to use the Learnosity item bank, which uses a flexible, tag-based system for organizing Items as well as providing test-construction facilities. Alternatively, clients can choose to simply add the Learnosity TEI editor to their existing Content Management System.

The Learnosity reporting solutions allow clients to easily embed HTML reports, with useful insights from individual and group analytics, on any webpage. Clients also have on-demand access to the raw data, providing the flexibility to use as needed at any time.


About A Pass Educational Group:

A Pass EducationA Pass Educational Group, LLC is very proud of its reputation for delivering high-quality work on time. We are also proud of our reputation as being responsive, friendly, and easy to work with. Finally, we cherish the loyalty of our staff and contract associates. We offer the following services: Instructional Design. A Pass recognizes that high-quality learning requires exemplary instructional design. Students and teachers must fully understand expectations and easily engage in content for optimum learning. Read More Assessment Design. A Pass develops many types of assessment questions, including multiple choice, true/false, constructed response, short essay, technology-enhanced items (TEIs) and more. We design assessments to accompany textbooks and study aids as well as high-stakes assessments for state and national standardized exams. Read More Course Development. The A Pass team can develop courses and /or lesson plans in any subject area. If we do not have the appropriate resources in house, we have the ability to find them. Read More Editing. A Pass has deep experience editing client content and resources—this includes developmental editing, copy editing, fact checking, and more. Read More



This post was posted in , , , by on