Phi Mu Alpha Sinfonia was founded by Ossian Everett Mills in 1885, and was originally called the Sinfonia Club. The fraternity began at the New England Conservatory of Music in Boston, Massachusetts, which would eventually become the Alpha chapter. As the years passed, the men of the Sinfonia Club began looking to expand their ideals to other worthy followers, and on October 6, 1898, now known as Founder's Day, a meeting was set to discuss expansion and the possible setup of an intercollegiate network. Two years later, Phi Mu Alpha Sinfonia officially became a fraternity with the admission of men at the Broad Street Conservatory in Philadelphia, Pennsylvania, founding Phi Mu Alpha’s Beta chapter.
While Phi Mu Alpha is now a social fraternity, this was not always the case. The professional period of Phi Mu Alpha began at the end of World War II and culminated in 1983. Men returning from war were less focused on the ideals of the organization and were instead seeking employment. The fraternity offered men training to improve their success as musicians, oftentimes specifically striving to improve their skills as music educators. As the years passed, members again began to focus more on the original ideals set forth by Ossian E. Mills and the other founding members of the fraternity.