This looks like it happened just after James and Lily were kissing and Sirius was like WHAT ARE YOU DOING!?!? IT'S QUIDDITCH! So James fell off his broom after being hit by a ball (can't remember what they're called..... bludgers?)
Book 5: In Snape's memory, James was playing with a Snitch, and according to Harry he had excellent reflexes, the reflexes of a Seeker. I literally just read through the books twice within the last few weeks, where does it say he's a Chaser?
"James was a brilliant student and was Quidditch Chaser on the Gryffindor house team. There is considerable confusion over whether James Potter was a Seeker or a Chaser. Rowling has confirmed that James was a Chaser, and that his being a Seeker was introduced in the Harry Potter films. Despite having been a Chaser, however, he was known to play with a golden snitch outside of games and practices, ostensibly to show off. He similarly had a habit of rumpling up his hair to produce an action-man image" - www.harrypotter.ca
I tried looking in the first book, can't really find it so I tried looking on Harry Potter wiki and it said "James was a skilled Quidditch player, playing Chaser for the Gryffindor Quidditch team during his school years. He was noted to have excellent reflexes, as seen when playing with a Snitch he'd taken from the Quidditch supplies."
According to Harry Potter wiki he was a chaser, but I'm sure in the books it says he was a seeker, I remember in PS Harry seeing the trophy which says he was a seeker, and in OotP in Snape's memory, James was playing with a snitch.
He actually was a chaser, JK said so in an interview. They made him a seeker in the movies, but that was wrong. It is never mentioned in the books. Also he's playing with a snitch in Snape's memory because he would have looked a lot less cool playing with a quaffle !