The house was built of iron-infused sandstone that was quarried from nearby Mount Airy plantation, the home of Rebecca Tayloe Lee.