Microsoft’s Bing AI made a few mistakes during its demo launch last week

During the chatbot hype last week, p Microsoft and Google trying to outdo each other in demoing early versions of AI-powered search, more than a million people signed up to try Microsoft’s tool in the first 48 hours, the company said.

Microsoft CEO Satya Nadella told CNBC that the technology, whichch can provide full answers that read as if they were written by a human, “maybe the industrial revolution led to knowledge work”.

But for those who care about accuracy, the AI ​​leaves much to be desired.

In a Microsoft demonstration to reporters, the ChatGPT-like technology built into the company’s Bing search engine analyzed revenue reports from Gap and Lululemon. When comparing its answers to the actual reports, the chatbot missed some numbers. Others were apparently made up.

“During the demo, Bing AI got some answers completely wrong. But nobody noticed,” independent search researcher Dmitry Brereton wrote in a Substack post on Monday. “Instead, everyone jumped on the Bing hype train.”

Brereton identified possible factual problems with Microsoft’s demonstration in his responses about the vacuum cleaner’s specifications and plans to travel to Mexico in addition to financial misstatements. He told CNBC that he wasn’t initially looking for bugs, but only discovered them when he took a closer look to write a comparison of Microsoft’s and Google’s AI.

Artificial intelligence experts call this phenomenon “hallucination,” or the tendency of tools based on large language models to just make things up. Last week, Google unveiled a rival AI tool that also contained factual errors, though those errors were quickly spotted by viewers.

Both companies are rushing to incorporate new kinds of generative artificial intelligence into search engines and are eager to show off their achievements after the explosion of ChatGPT, which OpenAI unveiled to the public in November. OpenAI has raised billions from Microsoft, while rival startups like Stability AI and Hugging Face have also grown to billions of dollars in private funding rounds.

While Google has been reluctant to add AI-generated answers to its search engines, citing reputational and security risks, Microsoft highlighted the short-term potential of releasing the technology to some of the public in its announcement last week.

“I think it’s important not to be in a lab,” Nadella said. “You have to get these things out safely.”

When it came time to demonstrate Bing AI’s response to a query about corporate earnings, there were some issues.

Yusuf Mehdi, Microsoft’s chief marketing officer, took to Gap’s investor relations site and asked AI Bing to summarize “key takeaways” from the retailer’s third-quarter earnings release in November.

“Very cool. Huge time saver,” Mehdi said.

These are screenshots from the Microsoft demo:

Here are some resume mistakes:

• Gap’s reported gross margin was 37.4%. But after excluding Yeezy-related charges, adjusted gross margin was 38.7%.
• Gap’s operating margin was 4.6%, not 5.9%, a number not found in the company’s report.
• Adjusted diluted earnings per share were $0.71 adjusted instead of $0.42, a figure not reported. The figure reported by Gap included an adjusted income tax of about $0.33.
• In August, Gap released its full-year outlook and said in its third-quarter report that “in the fourth quarter, net sales may decline by the mid-single digits compared to last year.” That would mean a full-year decline in revenue as opposed to “low double-digit growth.” There is no forecast for operating margin or earnings per share.

Microsoft said it is aware of the bugs and expects Bing’s AI to make mistakes.

“We are aware of this report and have analyzed its findings in our efforts to improve this experience,” a Microsoft spokesperson told CNBC. “We recognize that there is still work to be done and expect that the system may make mistakes during preview, so feedback is very important so that we can learn and help the models improve.”

Microsoft then asked Bing AI to compare Gap’s earnings with Lululemon’s report. Mehdi wanted Bing to compile the information from the two reports into a spreadsheet.

“Look at how amazing it is,” he said. “Just like that, in one table, I can get the answer to this question. Think how long it would have taken otherwise.’

Here’s what the Bing AI tool returned:

There are several errors in the table, starting with the fields.

• Lululemon’s gross margin was 55.9%, not 58.7%.
• The company’s operating margin was 19%, not 20.7%.
• Lululemon reported diluted earnings per share of $2 and adjusted earnings per share of $1.62. Bing reported a diluted earnings per share figure of $1.65.
• Gap had $679 million in cash and cash equivalents, not $1.4 billion.
• Gap’s inventory was $3.04 billion, not $1.9 billion.

WATCH: Full CNBC interview with C3.ai CEO Thomas Siebel