In classrooms, teachers put AI tutoring bots to the test

On a recent morning, Cheryl Drakeford, a third-grade teacher at First Avenue Elementary School in Newark, projected a challenging math question onto her classroom whiteboard: “Which part of the letters in the word MATHEMATIC are consonants?”

Mrs. Drakeford knew that “consonant” might be an unfamiliar word to some students. So she suggested asking Khanmigo, a new tutoring bot that uses artificial intelligence, for help.

She paused as about 15 schoolchildren dutifully typed the same question: “What are consonants?” – in their math software. Then she asked the third graders to share the answer from the tutoring bot.

“Consonants are the letters in the alphabet that are not vowels,” a student read aloud. “The vowels are A, E, I, O and U. Consonants are all other letters.”

The hype in the tech industry and doomsday scenarios surrounding AI-enhanced chatbots like ChatGPT led many schools this year to push to block or restrict the use of the tools in classrooms. Newark Public Schools takes a different approach. It is one of the first school systems in the United States to pilot test Khanmigo, a computerized learning tool developed by Khan Academy, an education nonprofit whose online classes are used by hundreds of districts.

Newark has essentially volunteered as a guinea pig for public schools across the country trying to differentiate the practical uses of new AI-assisted tutoring bots from their marketing promises.

Proponents argue that classroom chatbots could democratize the idea of ​​tutoring by automatically adjusting answers to students, allowing them to work on lessons at their own pace. Critics warn that the bots, trained on massive databases of texts, can fabricate plausible-sounding misinformation, making them a risky bet for schools.

Officials in Newark, the largest district in New Jersey, said they were tentatively testing the tutoring bot at three schools. Their findings could impact districts across the United States that will vet AI tools this summer for the upcoming school year.

“It is important to introduce our students to it, because it will not go away”, Timothy Nellegar, the director of educational technology at Newark Public Schools, said of AI-assisted technology. “But we need to figure out how it works, the risks, the good and the bad.”

Khan Academy is one of the few online learning companies to have created new tutoring bots based on language models developed by OpenAI, the research lab behind ChatGPT. Khan Academy, whose high-level tech donors include Google, the Bill and Melinda Gates Foundation, and the Elon Musk Foundation, was granted access to the AI ​​models last year.

The tutoring bot is designed specifically for schools and often walks students through the sequential steps required to solve a problem.

When the Khan Academy started looking for districts to test its experimental tutorbot this spring, Newark volunteered. A number of local primary schools are already using the educational organization’s online math lessons to help them maintain mastery of concepts such as grouping numbers. And the AI ​​tool would be free for those schools during the initial testing phase.

District officials said they wanted to see if Khanmigo could improve student engagement and math learning. Schools like First Avenue, which many children from lower-income families attend, were also eager to give their students an early opportunity to try out a new AI-assisted learning tool.

Districts such as Newark that use Khan Academy’s online classes, analytics and other school services – not including Khanmigo – pay an annual fee of $10 per student. Participating districts that want to test Khanmigo for the upcoming school year will pay an additional $60 fee per student, the nonprofit said, noting that the computing costs for the AI ​​models were “significant.”

Newark students started using Khan’s computerized learning tool in May. The reviews have been mixed so far.

On a recent morning, sixth graders at First Avenue Elementary were working on a statistics assignment that involved developing their own consumer surveys. Their teacher, Tito Rodriguez, suggested that students start by asking Khanmigo two background questions: What is a survey? What makes a question statistical?

Mr. Rodriguez described the bot as a helpful “co-teacher” that allowed it to spend extra time with kids in need of tutoring while allowing more self-directed students to plow ahead.

“Now they don’t have to wait for Mr. Rodriguez,” he said. “They can ask Khanmigo.”

Later in Ms. Drakeford’s math class, the bot’s responses to students sometimes seemed less like suggestions and more like direct answers.

When students asked Khanmigo the fraction question posted on the whiteboard in class, the bot replied that the word “mathematician” contained 13 letters and that seven of those letters were consonants. That meant the fraction of consonants was seven out of 13, the bot wrote, or 7/13.

“That’s our biggest concern, that too much of the thinking goes through Khanmigo,” he said Alan Usherenko, the district’s special assistant for schools including First Avenue, in Newark’s North Ward. The district didn’t want the bot to walk students through a problem step-by-step, he said, adding, “We want them to know how to tackle the problem themselves, to use their critical thinking skills.”

In an email, Khan Academy said students often needed initial support as they progressed through problem-solving steps, and practice could help them progress through the steps automatically, without assistance.

The group added that the tutoring bot is designed to help students solve problems, not give them the answers. But in the case of Newark’s fracture problem, the organization said, Khanmigo “helped too much, too quickly.”

“Our engineering team corrected the AI ​​a few weeks ago,” Khan Academy said in an email Tuesday, “so that it no longer provides the answer to this question.”

On Wednesday, a reporter asked Khanmigo the same split question. In student mode, the tutoring bot explained the steps and then immediately gave the answer: “the fraction of consonants in the word ‘MATHEMATIC’ is 7/13.”

In teacher mode, which is designed to guide teachers through problems and answers, the bot gave a different — incorrect — answer. Khanmigo falsely said that there were eight consonants in the word ‘mathematician’. That led to the bot giving a wrong answer: “8 consonants / 14 total letters = 8/14”

In an email, Khan Academy said it resolved the issue in its “tutor me: math and science” section for students, noting that the reporter had asked a question in another section of the site. “As for the teacher mode giving the wrong answer,” the email said, “sometimes Khanmigo makes mistakes.”

Still, Mr. Usherenko said he was hopeful. The district had suggested to Khan Academy that instead of relying on students to ask Khanmigo the right questions, it would be more helpful if the bot asked students open-ended questions and analyzed their answers.

“It’s not where I want it yet,” Mr Usherenko said of Khanmigo. “But if it can find students’ misconceptions, that will be a game changer.”

Khan Academy said the tutoring bot often asked students open-ended questions and the group worked to get the AI ​​models to accurately identify misconceptions. The nonprofit added that it continued to improve Khanmigo with feedback from school districts.

Whether schools can afford AI-assisted tutorbots remains to be seen.

Khan Academy said it would offer a discount for districts where more than half of students qualified for a free or reduced-price lunch. Still, the financial hurdles suggest that AI-enhanced classroom chatbots are unlikely to democratize education any time soon.

Mr. Nellegar, Newark’s ed tech director, said his district was looking for outside funding to help cover Khanmigo’s costs this fall.

“The long-term cost of AI is a concern to us,” he said.