Dolley Payne (also spelled Dolly) was born on May 20, 1768, in New Garden, North Carolina. Soon after, her family moved to Virginia. During her childhood, she was a Quaker, but when she married James Madison her religion changed. She had two older brothers, Walter and William Temple. There were five younger children, Isaac, Lucy, Mary, Anna, and Johnny. Dolley did not believe in slavery and hated fighting. When she was seven her family moved to Scotchtown where she heard many ghost stories but did not believe in any of them.
She loved to garden and ride horses, but she didn't spell well. Dolley was always optimistic and loved adventure. At an early age, she was taught to sew, cook, and nurse the sick. When she was twelve, she made her first trifle, a very sweet dessert with cream inside, and Mr. Payne said that it was the best trifle that mother had ever made. Mother replied, "Your eldest daughter made it."