Philly Joe Jones was born with the name Joseph Rudolph Jones in the city of Philadelphia on July 15, 1923. His mother, a piano teacher taught him the basics in music. In his formative years he also studied the drums with drummers the likes of Cozy Cole and Charles Wilcoxon, receiving valuable advise from Art Blakey and a then younger Max Roach. He established himself as "Philly Joe" Jones, from the name of the city of his birth, to distinguish himself from the mainstay Count Basie’s drummer, Jo Jones. But just as Jo Jones established the rhythm section standard in the 30’s and 40’s, Philly Joe would do the same in the 50’s.

He began playing with the rhythm and blues bands in the 40’s, establishing himself on the New York jazz scene. The first group that he recorded with was co-led by Johnny Griffin and Joe Morris, along with Matthew Gee on trombone and Elmo Hope on piano and Percy Heath. For a time they played standard Monk and pieces of Bud Powell, but reverted back to playing the blues for the sake of money. Philly, unhappy with the music, split with the group and joined the already established Ben Webster, Lee Konitz, Zoot Sims, Tony Scott, and Tadd Dameron. He worked in Philadelphia as the local drummer for stars such as Dexter Gordon and Fats Navarro. He was in the army for a short time and len left in 1945 taking a job up as a streetcar driver. He was supposedly fired from the job because he would stop the steetcar with people in it and go in to play a set at the jazz clubs on the way, sometimes forgetting about the people in the car.

The two major influences that have shaped Philly Joe’s career was his work with Tadd Dameron in the late 40’s to 1953 and then with Miles Davis from 1952-58. Working with Dameron taught him to play as a ‘big band’ drummer as well as behind a soloist. The most important association was that with trumpeter Miles Davis. The two would travel around the US stopping in cities to do a gig with the local talent. "Philly Joe Jones and I would go from city to city playing with local musicians. Philly would go ahead of me and get some guys together and then I would show and we’d play a gig. But most of the time this shit was getting on my nerves because the musicians didn’t know the arrangements and sometimes didn’t even know the tunes"(Miles’ Autobiography 179). It became "a drag" as Philly said because usually the local talent was not close to satisfactory. This went on from 52-55. Then after a booking at a club in Baltimore, 1955, after being found by Philly Joe, Miles hired pianist Red Garland from Texas and bassist virtuoso Paul Chambers, then of 20 years of age. An invitation was extended to John Coltrane, who was tested by a the Miles arrangement of Two Bass Hit. After the gig Miles said to Jones, "I think this is it." Jones agreed having said of the group, "The first time we played together…we just looked around at each other and said, ‘hum here it is right here. We’ve got musical telepathy here. We have five people who always know what’s going to happen next.’" And those five people became legendary as the classic Miles Davis Quintet was baptized for its first time.

Chambers and Jones would be the dynamic rhythm section popular among soloists such as Sonny Rollins and Art Pepper. Jones was on many Blue Note recordings and was virtually house drummer for the Blue Note, Riverside, and Prestige record labels. His style has been described as "too loud" or "like a machine gunner." However one must listen carefully to appreciate his precision and energy. One excellent example of Jones’ playing can be heard of Coltrane’s album, Blue Train. Jones’ first album was in 1958, an LP for the Riverside record label. On his album were players that would become well known in the business: Nat Adderley, Julian Priester, Johnny Griffin and Tommy Flanagan. In his prime, Jones won great recognition. In Downbeat magazine polls he won the new star category on drums in 1957 and the main drum category in 1962. He was second place in 1961, second to Max Roach.

In 1962 there was a sharp decline in Jones’ career. He was ignored and almost forgotten. One reason was a drummer named Elvin Jones with John Coltrane’s quartet, who stole some of the limelight. Perhaps this the reason that Jones moved to England in 1968. There he made his mark as a teacher, tremendously improving the local talent. He eventually returned to the New York and west coast jazz scene, however lost from the forefront of it all. He did work regularly in his home town of Philadelphia. In 1977 he cut an LP for the Sonet label: Mean What You Say, an good example of latter-day bebop.

Jones, in the 80’s, formed Dameronia, a group dedicated to playing original recordings of Tadd Dameron’s music. This resulted in two Uptown label LPs in 82 and 84 which were well received by critics. However on August 30th, 1985, his electrifying solos would be no more as he passed away of a heart attack in his home in "Philly."