The de Hoo family lived in a manor house on the site in the 13th century. The houses on the site changed hands several time until 1762 when owner Francis Hearne sold the estate to John Stuart, 3rd Earl of Bute.

After his Prime Minister years from 1762 to 1763, Stuart focused his energies on the his estate at Luton Hoo. Robert Adam, a neoclassical architect built the present house for the 3rd Earl of Bute. He began work in 1767. The building was halted by a fire in 1771. By 1774 the house was occupied even though it was not finished. Luton Hoo was one of the largest houses for which Adam was wholly responsible. The mansion was transformed by architect Robert Smirke about 1830 following the occupation of the 3rd Earl's grandson who was the 2nd Marquess of Bute. He redesigned the house except for the south front.

A fire in 1843 destoryed much of the house and its contents. The house was sold to John Leigh in 1848, who rebuilt the shell in the same style and manner of Smirke. The Leigh family owned the house until the death of John Leigh's daughter in law in 1903.