This looks like it happened just after James and Lily were kissing and Sirius was like WHAT ARE YOU DOING!?!? IT'S QUIDDITCH! So James fell off his broom after being hit by a ball (can't remember what they're called..... bludgers?)
Book 5: In Snape's memory, James was playing with a Snitch, and according to Harry he had excellent reflexes, the reflexes of a Seeker. I literally just read through the books twice within the last few weeks, where does it say he's a Chaser?
"James was a brilliant student and was Quidditch Chaser on the Gryffindor house team. There is considerable confusion over whether James Potter was a Seeker or a Chaser. Rowling has confirmed that James was a Chaser, and that his being a Seeker was introduced in the Harry Potter films. Despite having been a Chaser, however, he was known to play with a golden snitch outside of games and practices, ostensibly to show off. He similarly had a habit of rumpling up his hair to produce an action-man image" - www.harrypotter.ca