Grofé left home at age 14 and variously worked as a milkman, truck driver, usher, newsboy, elevator operator, helper in a book bindery, iron factory worker, and as a piano player in a bar for two dollars a night and as an accompanist. He continued studying piano and violin. When he was 15 he was performing with dance bands. He also played the alto horn in brass bands. He was 17 when he wrote his first commissioned work.
Grofé's most memorable arrangement is that of George Gershwin's Rhapsody in Blue, which established Grofé's reputation among jazz musicians. Grofé took what Gershwin had written for two pianos and orchestrated it for Whiteman's jazz orchestra. He transformed Gershwin's musical canvas with the colors and many of the creative touches for which it is so well known. He went on to create two more arrangements of the piece in later years. Grofé's 1942 orchestration for full orchestra of Rhapsody in Blue is the one most frequently heard today. In 1928, George Gershwin wrote a letter to ASCAP complaining that Grofé had listed himself as the composer of Rhapsody in Blue. In spite of this misunderstanding, Grofé served as one of the pallbearers at Gershwin's funeral in 1937.
In 1932, The New York Times called Grofé "the Prime Minister of Jazz". This was an oblique reference to the fact that Whiteman was widely called "King of Jazz", especially after the appearance of the 1930 film of that name which featured Whiteman's music.
Due to Grofé's ubiquity in arranging large-scale musical works and a perceived paucity of American achievements in serious music, the German conductor Wilhelm Furtwängler complained that "America has no composers, only arrangers."
During this time, Grofé also recorded piano rolls for the American Piano Company (Ampico) company in New York. These captured performances were embellished with additional notes after the initial recording took place to attempt to convey the thick lush nature of his orchestra's style. Hence the published rolls are marked "Played by Ferde Grofé (assisted)".
Not everybody appreciated Grofé's flowery arrangements during this time. In a review of a Whiteman jazz concert in New York, one writer said the music was expected to be pleasing, and "it proved so when it was repeated last night, in spite of the excessive instrumentation of Ferde Grofé." A writer of a later generation said "the Grofé and Gould pieces were the essence of slick commercialism..."
In 1943, he was a guest on Paul Whiteman Presents. During the 1930s, he was the orchestra leader on several radio programs, including Fred Allen's show and his own The Ferde Grofé Show. The "On the Trail" segment of Grand Canyon Suite was used for many years as the "musical signature" for radio programs sponsored by Philip Morris cigarettes, beginning with their 1933 program featuring Grofé and his orchestra. Jon Hendricks wrote lyrics for "On the Trail", and the song was recorded for Hendricks' album To Tell the Truth (1975).
Several times he conducted orchestral programs in New York's Carnegie Hall. In January 1933 the premiere of his Tabloid, an orchestral suite in 4 movements, was presented in Carnegie Hall. In 1937, he conducted a concert tribute to George Gershwin at Lewisohn Stadium. Turnout (20,223 people) was the largest in that stadium's history.
In 1934, Grofé announced that he was working on an opera, to be based on the Edgar Allan Poe story "The Fall of the House of Usher".