Spin-flop ordering from frustrated ferro- and antiferromagnetic interactions: A combined theoretical and experimental study of a Mn/Fe(100) monolayer