This looks like it happened just after James and Lily were kissing and Sirius was like WHAT ARE YOU DOING!?!? IT'S QUIDDITCH! So James fell off his broom after being hit by a ball (can't remember what they're called..... bludgers?)
Book 5: In Snape's memory, James was playing with a Snitch, and according to Harry he had excellent reflexes, the reflexes of a Seeker. I literally just read through the books twice within the last few weeks, where does it say he's a Chaser?