The idea of a railway museum in France dates back to the early 20th Century, when the Universal Exhibition in Paris was closed. A little later, in 1944, it was suggested that different sites were used to open such a museum in Paris, the Grand Palais, the Gare Montparnasse, the Champ de Mars station or the train station of Les Invalides. But many difficulties did not allow for completion of this project. The French National Railways (SNCF) could hardly take much interest in the development of their "past" because their pûrpose was to put energy into their future.
In 1961the Directorate of equipment and traction decided to accumulate the material preserved in a former warehouse in Chalon-sur-Saône. In the same year, under the advice of Jean-Mathis Horrenberger, the industrial society of Mulhouse created a committee that was responsible for studying the possibility of constructing a railway museum in Mulhouse on the land that the city offered. The project was finally accepted in 1969 by the SNCF.
In this same year it was also decided that the Association of the French Railway Museum would be formed, with the mission of creating a museum and managing it.
At the beginning of the year 1971, the first locomotive parked in Chalon-sur-Saone arrived at Mulhouse to be exhibited there temporarily in the pavilion of a former SNCF camp in North-Mulhouse.