New York City’s MyCity ChatBot was introduced as a pilot meant to help business owners move faster through government information. But testing reported by The Markup and The City found that the tool can give incorrect answers on issues where precision matters, including housing rules, worker pay, workplace protections, and industry-specific compliance.
The problem is not simply that a chatbot made a mistake. The concern is that a government-run tool can sound authoritative while giving advice that conflicts with city policy, and users may not know when they need to verify the answer elsewhere.
A chatbot built to guide business owners
NYC’s “MyCity” ChatBot launched as a “pilot” program last October. The city’s announcement described it as a way for business owners to “save … time and money by instantly providing them with actionable and trusted information from more than 2,000 NYC Business webpages and articles on topics such as compliance with codes and regulations, available business incentives, and best practices to avoid violations and fines.”
That positioning matters. A tool framed around official NYC Business information is not the same as a casual experiment. When a business owner asks about fines, regulations, pay rules, or compliance, the answer can shape real-world decisions.
The chatbot is also prominently labeled as a “Beta” product. Its warnings say it “may occasionally produce incorrect, harmful or biased content” and that users should “not rely on its responses as a substitute for professional advice.” At the same time, the page says it is “trained to provide you official NYC Business information” and presents the tool as a way “to help business owners navigate government.”
Those two messages sit in tension. A warning tells users to be cautious, while the product pitch encourages them to treat the chatbot as a guide to government information.
Where the answers went wrong
The Markup and The City found the MyCity chatbot giving wrong information about basic city policies. One example involved Section 8 housing vouchers. The bot said that NYC buildings “are not required to accept Section 8 vouchers.”
That answer conflicts with an NYC government info page cited in the source article, which says Section 8 housing subsidies are among the lawful sources of income that landlords must accept without discrimination.
The incorrect responses were not limited to one housing question. The Markup also received wrong information when asking about worker pay and work hour regulations. It also found problems in industry-specific areas, including funeral home pricing.
Further testing from BlueSky user Kathryn Tewson showed the chatbot giving wrong answers about workplace whistleblowers. The same testing also surfaced bad answers about the need to pay rent.
For a user, the practical risk is straightforward: a confident answer can look useful even when it is unreliable. If the answer concerns rent, lawful income, wages, hours, or business obligations, the cost of relying on it can be serious.
Why generative AI can fail this way
The source article ties the issue to how these chatbots work. MyCity’s chatbot is powered by Microsoft Azure and uses token-based predictive models. In plain terms, the system relies on statistical associations across millions of tokens to predict likely next words in a sequence.
That approach can produce fluent answers without real understanding of the information being communicated. The output can sound complete and official even when the underlying answer is wrong.
The Section 8 example also shows that the same question may not always produce the same result. The Markup said at least one test produced the correct answer on the question about accepting Section 8 housing vouchers, while “ten separate Markup staffers” received the incorrect answer when repeating the same query.
That inconsistency is a major challenge for government services. Users do not only need an answer that sounds plausible. They need the correct answer, especially when the topic is law, regulation, or municipal policy.
Officials and business advocates respond
Andrew Rigie, executive director of the NYC Hospitality Alliance, told The Markup that he had encountered inaccuracies from the bot himself. He also said he had received reports of similar issues from at least one local business owner.
NYC Office of Technology and Innovation Spokesperson Leslie Brown defended the tool’s usefulness while acknowledging continued work on it. Brown told The Markup that the bot “has already provided thousands of people with timely, accurate answers” and that “we will continue to focus on upgrading this tool so that we can better support small businesses across the city.”
That response points to the broader tradeoff. A chatbot can make information easier to access, but the value depends on whether users can trust what it says. In a government setting, speed is not enough if the answer can steer people away from the rules they are required to follow.
A warning for public AI tools
The MyCity case fits a wider pattern described in the source article: public-facing chatbots are being deployed before their accuracy and reliability have been fully vetted. The article cites a court forcing Air Canada to honor a fraudulent refund policy invented by a chatbot on its website. It also cites a Washington Post report finding that chatbots in major tax preparation software provided “random, misleading, or inaccurate … answers” to many tax queries.
The source article also notes reports of car dealership chatbots being tricked into accepting a “legally binding offer – no take backsies” for a $1 car.
These examples differ in context, but they point to the same core issue. When a chatbot speaks on behalf of an organization, its mistakes can create confusion, liability, or harm. That risk becomes sharper when the organization is a government and the questions involve public rules.
The article also notes that some companies are moving away from generalized LLM-powered chatbots toward more specifically trained Retrieval-Augmented Generation models, tuned only on a smaller set of relevant information. The source frames that narrower focus as potentially more important if the FTC succeeds in efforts to make chatbots liable for “false, misleading, or disparaging” information.
For now, the MyCity episode is a reminder that AI tools used for public service need more than a friendly interface. They need reliable boundaries, clear verification paths, and accuracy standards that match the stakes of the questions people are asking.