After the Langhornes left, Point of Honor was sold by Judge Daniel in 1862 to Col. Robert Latham Owen and his family. Owen was the president of the Virginia and Tennessee Railroad that ran westward from Lynchburg to Bristol, Virginia on the Tennessee border. During the Civil War, the Virginia and Tennessee Railroad was one of the major arteries transporting troops and supplies for the Confederacy.
Throughout the Civil War the hospitality of Point of Honor was extended to the many Confederate officials and army officers calling on business with Col. Owen. On the eve of the Battle of Lynchburg in June 1864, Mrs. Narcissa Owen recalled how she welcomed two Confederate stragglers onto the porch. After offering what food and refreshment she could, Mrs. Owen attempted to reassure the pair by telling them, the name of every Confederate officer she knew and of all the reinforcements they would be bringing to defend the city. The stragglers were Federal army spies and they immediately reported Mrs. Owen's exaggerations to General Hunter, who withdrew from Lynchburg, possibly convinced that he was heavily outnumbered.