In video circulating on social media, racegoers can be seen brawling in front of horrified spectators inside the main grandstand building, leaving one man covered in blood.
Officials have said that anti-social behaviour “has no place” at Epsom.
Six people have been arrested on suspicion of grievous bodily harm and affray and released pending further investigation.
Surrey Police said officers were called to the incident just before 6pm on Saturday, with witnesses claiming two men fought for several minutes despite attempts by staff and bystanders to intervene.
Video of the incident shows one man having his white shirt ripped from his body before he falls to the ground, then getting back up and continuing to throw punches.
The brawl then spilled to the ground, with another racegoer being knocked to the floor amid the chaos. One of the men is later seen with blood covering much of his face and shirt.
A witness said: “The response team and a few people were trying to split them up but they were just going at it for four or five minutes before the police got there.”
A spokesperson for Epsom, which is owned by Jockey Club Racecourses, said: “Our security teams and police on site responded quickly to an incident on Saturday.
“This matter is now being dealt with by Surrey Police so it would not be appropriate to comment further, other than to say that this sort of behaviour has no place at our racecourse.”
Epsom Derby Day is one of the highlights of the British sporting calendar, attracting tens of thousands of visitors every year.
Arrests are not new to the event however - last year, four arrests were made across the two-day meeting, while 10 were arrested the previous year.
A crowd of 28,500 on the Surrey Downs on Saturday saw Christmas Day claim Derby glory amid wet and windy weather, including the King and Queen who made the trophy presentation to winning trainer Aidan O’Brien, jockey Ronan Whelan and owners Coolmore.