A set of words is factorially balanced if the set of all the factors of its words is balanced. We prove that if all words of a factorially balanced set have a finite index then this set is a subset of the set of factors of one Sturmian word. Moreover, characterizing the set of factors of a given length n of a Sturmian word by the left special factor of length n-1 of this Sturmian word, we provide an enumeration formula for the number of sets of words that correspond to some set of factors of length n of a Sturmian word.