Owen sold Point of Honor to Seth Woodruff and William Spence, partners in banking and brokerage. Neither apparently resided in the mansion and it is assumed that the house was rented. In 1876, the property was subdivided and purchased by T. V. Strange and L. E. Lichford, and by 1878 Lichford is listed as the sole owner of the mansion. Lichford was a grocery wholesaler and had a business warehouse near Point of Honor. His family occupied the house for over fifty years, three generations calling the old Cabell mansion home.
Alterations were made at various times to modernize the house and to enlarge it to a certain extent. Indoor plumbing was installed and several additions were made in the rear of the house and the side porch was enclosed to add another bathroom.
Through the years, the outbuildings and gardens disappeared; new houses were built in the rear yard and along the old narrow road (Norwood Street), finally surrounding the house and removing the rural atmosphere of a century before. In the 1920s, the Lichfords, like so many others, decided to move to the Boonsboro Road area, which was developing rapidly. They built a spacious new home on Peakland Place and put Point of Honor up for sale. In 1928 James R. Gilliam, Jr., recognizing the historical value of the mansion, purchased Point of Honor and deeded it to the City of Lynchburg. Point of Honor ceased to be a residence and became a recreation center for the Daniel’s Hill neighborhood. This proved to be its salvation.