Charles Warman Roberts married Annie Edensor Marsden (1824-1895) in Sydney in June 1845. They had eight children (five sons and three daughters); their eldest son, Charles James Roberts (1846-1925), became a member of parliament, served as Mayor of Sydney and was NSW Postmaster General from 1884 to 1889. Charles and Annie lived at 'Fernleigh' at Rose Bay between 1874 and 1892 before moving to a house named 'Idalia' in Paddington.