James Vernon Taylor was born at Massachusetts General Hospital in Boston on March 12, 1948, where his father, Isaac M. Taylor, worked as a resident physician. His father came from a wealthy Scottish family from the South, however, part of his ancestral roots are deep in Massachusetts Bay Colony and include Edmund Rice, one of the founders of Sudbury, Massachusetts. His mother, the former Gertrude Woodard (1921–2015), studied singing with Marie Sundelius at the New England Conservatory of Music and was an aspiring opera singer before the couple's marriage in 1946. James was the second of five children, the others being Alex (1947–1993), Kate (born 1949), Livingston (born 1950), and Hugh (born 1952).

In 1951, his family moved to Chapel Hill, North Carolina, when Isaac took a job as an assistant professor of medicine at the University of North Carolina School of Medicine. They built a house in the Morgan Creek area off the present Morgan Creek Road, which was sparsely populated. James would later say, "Chapel Hill, the Piedmont, the outlying hills, were tranquil, rural, beautiful, but quiet. Thinking of the red soil, the seasons, the way things smelled down there, I feel as though my experience of coming of age there was more a matter of landscape and climate than people." James attended public primary school in Chapel Hill. Isaac's career prospered, but he was frequently away from home, on military service at Bethesda Naval Hospital in Maryland, or as part of Operation Deep Freeze in Antarctica in 1955 and 1956. Isaac Taylor later rose to become dean of the UNC School of Medicine from 1964 to 1971. Beginning in 1953, the Taylors spent summers on Martha's Vineyard.

James first learned to play the cello as a child in North Carolina and switched to the guitar in 1960. His guitar style evolved, influenced by hymns, carols, and the music of Woody Guthrie, and his technique derived from his bass clef-oriented cello training and from experimenting on his sister Kate's keyboards: "My style was a finger-picking style that was meant to be like a piano, as if my thumb were my left hand, and my first, second, and third fingers were my right hand." He began attending Milton Academy, a preparatory boarding school in Massachusetts in fall 1961. Summering before then with his family on Martha's Vineyard, he met Danny Kortchmar, an aspiring teenage guitarist from Larchmont, New York. The two began listening to and playing blues and folk music together, and Kortchmar quickly realized that Taylor's singing had a "natural sense of phrasing, every syllable beautifully in time. I knew James had that thing." Taylor wrote his first song on guitar at 14, and he continued to learn the instrument effortlessly. By the summer of 1963, he and Kortchmar were playing coffeehouses around the Vineyard, billed as "Jamie & Kootch".

Taylor faltered during his junior year at Milton, feeling uneasy in the high-pressure college prep environment despite good scholastic performance. The Milton headmaster would later say, "James was more sensitive and less goal-oriented than most students of his day."