What do grad schools base their admissions on?
They don’t have many things to go on. The two main ones are course grades (along with course choice/difficulty) and recommendation letters. This is why rec letters are so important; they’re roughly 50% of the info the schools have on you. A third item is GRE scores (both the Physics and the General), but different schools use these to varying degrees, and not nearly as much as grades and rec letters.