This has already been tried in at least one court.
There was that story a while back about the guy who was told by an airline's AI help-desk bot that he would get a ticket refund if turned out he was unable to fly, only for the airline to say they had no such policy when he came to claim.
He had screenshots and said he wouldn't have bought the tickets in the first place if he had been told the correct policy. The AI basically hallucinated a policy, and the airline was ultimately found liable. Guy got his refund.
And the airline took down the bot.