Other sports backing the protest include softball, equestrian, handball, volleyball, triathon, fencing and wrestling.
Since taking over from Juan Antonio Samaranch in 2001, Rogge has made it clear that he wants to downsize the Games.
The former Olympic yachtsman has refused to allow any new sports into the Games until others are dropped. By suggesting the controversial votes, Rogge has made himself unpopular among many of the sports federations chiefs, some of whom are also IOC members.
The new row comes after the Belgian surgeon was accused earlier this week of being too harsh on the bidding cities for the 2012 Games after he called an investigation into London's £15million "incentives and sweeteners" package for athletes and sports bodies.