Dufton Hall stands on the north side of the village, and is in the form of the letter L. The Hall, which dates at least from the seventeenth century and possibly the sixteenth, is thought to have been the site of the medieval Manor House. In the early part of the thirteenth century the manor belonged to the Greystokes, from whom it passed in marriage to the Dacres of Gilsland.
Afterwards it was owned by Philip Howard, Earl of Arundel, whose grandson, Henry, granted a lease of the lordship for ninety-nine years to Sir Christopher Clapham in the seventeenth century. Clapham took advantage of an omission in the lease and cut down and sold the whole of Dufton Wood, making more from the timber than he had paid for all of Dufton. The lordship was subsequently purchased by John Winder, Esq. and in 1785 it was sold to the Earl of Thanet. The southern frontage was rebuilt in 1779 and is now of three storeys. The building was occupied as a sporting seat by Sir William Bryan Cooke in 1829 but since then it has been divided into tenements and today comprises four separate households.