The original owners of the house were Carrie and Colonel James Henry Rumbough. Carrie was the daughter of Joseph Powell, a Federal Circuit Judge from South Carolina. He was the only member of the South Carolina legislature to vote against the secession from the Union. Colonel Rumbough’s parents were Jacob Rumbough, from Woodstock, Virginia, and Ann Danridge, who was a descendant of Martha Washington.
Jacob was a railroad contractor, who supervised the building of the railroad from Lynchburg, Virginia to Greenville, Tennessee, where he ultimately settled.
James and Carrie Rumbough moved from Greeneville, Tennessee, where James had run a stagecoach operation that ran from Greeneville, Tennessee to Greenville, South Carolina by way of Warm Springs (now Hot Springs). When James decided to join the Confederate Army, he wanted his family to have a safe place to live since he didn't know if he would survive the war. So they moved to Warm Springs.
In 1866, four tracts of land south of the French Broad River, including the entire town of Warm Springs, was sold to Carrie Rumbough by Joseph A. McDowell of Buncombe County.
In 1868, Rutland (now the Mountain Magnolia Inn) was built for Carrie and the children to live in. At this time, Colonel Rumbough was away at war, and Carrie was running the Patton Hotel, which was located by the hot springs. The hotel, which burned in 1884, accommodated about 300 people, and attracted many visitors to the springs and the area.