Goodman was intelligent, unassuming, happy, and outgoing. He grew up as the second of three sons in a liberal household on the Upper West Side of Manhattan. Visitors to the Goodman home included Alger Hiss, blacklisted actor Zero Mostel, and the attorney who represented the Hollywood Ten, Martin Popper. (Popper became the Goodman family spokesman following Andy's disappearance.) Goodman's family was well-off, and he enjoyed summers at a lake home in the Adirondacks.
Goodman attended the progressive Walden School, widely known for its anti-authoritarian approach to learning. While a high school sophomore at Walden, Goodman travelled to Washington, D. C. to participate in the "Youth March for Integrated Schools." As a senior, he and a classmate visited a depressed coal mining region in West Virginia to prepare a report on poverty in America.
After graduating from Walden, Goodman enrolled at Queens College in part because of its strong drama department. Soon, however, his longing for commitment led him away from his interest in drama and back to politics. In April 1964, Goodman listened to a speech by Allard Lowenstein outlining a bold strategy for bringing civil rights to Mississippi. Lowenstein described Mississippi as "the most totalitarian state in America," a feudal backwater where racism was woven into the fabric of society. If the battle for civil rights could be won in the heart of the resistence, it could be won anywhere. Goodman applied for and was accepted into the Mississippi Summer Project. Although not seeing himself as a professional reformer, Goodman knew that his life had been somewhat sheltered and thought that the experience would be educational and useful.