He was repeatedly accused of having an affair with his sister-in-law
Dickens had just about the strangest relationship with his wife’s family, and it made people uncomfortable. He was so devoted to them that when one of his sisters-in-law passed away at 17, he wrote to a friend that he wished to have been buried with her. Not creepy at all.
It was his wife’s other sister that caused all the scandal though. Georgina was more devoted to Dickens then her sister was, making him the centre of her life. She lived in his home as a housekeeper and advisor, and repeatedly turned down marriage proposals to stay in the house with Charles.
When Dickens started having trouble with his wife, the public blamed Georgina, claiming that they were having an affair (which technically was considered incest in those Victorian times), and that she was secretly the mother of 3 of his children.
It was all rumours, however, but Dickens went overboard in order to prove it. He forced Georgina to get her virginity checked and shared the results publicly.