Virginia Gilmore was born Sherman Virginia Poole in El Monte, California. Her father was a retired officer of the British Army. Gilmore began her stage career in San Francisco at the age of 15, but moved to Los Angeles in 1939 to pursue work in films.