The far-right British National Party won its first ever seat in the European Parliament, while the anti-European United Kingdom Independence Party (Ukip) looked set to win more support than Labour in a national count.
With 20 out of 72 parliamentary seats declared, the BBC projected Labour would win just 16 percent, behind the main opposition Conservatives on 27 percent and Ukip -- which wants Britain out of the EU -- on 17 percent.
In another symbolic defeat, Labour was forced into second place in Wales for the first time since 1918, conceding pole position to the Conservatives.
The results are disastrous for Brown, who is fighting for his political life after a turbulent week that saw 10 ministers resign, a Labour collapse in local elections and intense speculation about a plot within Labour to oust him.
Labour deputy prime minister Harriet Harman had said she was expecting "very dismal results", saying voters were "absolutely furious" with the ruling party following revelations about the lavish expenses claims of lawmakers.
The scandal -- reinforced by the deep recession and rising unemployment -- was expected to benefit fringe parties such as the BNP, which won its first seat in the EU parliament in the Yorkshire and the Humber region in northern England.
Andrew Brons came fourth with almost 10 percent of the vote, taking a seat from Labour in what BNP chairman Nick Griffin said was a "huge breakthrough".
Mainstream politicans revile the BNP for its anti-immigration stance -- it wants to "voluntarily resettle" all immigrants to their countries of origin -- and Brown has called them a "nasty extreme" party.
Griffin said after the victory: "We are not a racist party. We do say this country is full up. The key thing is to shut the door."
In an interview with Sky News, he reiterated his party's anti-Islam stance, saying: "This is a Christian country and Islam is not welcome, because Islam and Christianity, Islam and democracy, Islam and women's rights do not mix.
"That's a simple fact that the elites of Europe are going to have to get their heads round and deal with over the next few years."
Griffin was bidding for a seat in northwest England, and as he arrived for the count in Manchester late Sunday several dozen protesters holding placards proclaiming them as a "Nazi" party tried to stop him entering the building.
Shahid Malik, MP for Dewsbury in Yorkshire who was Britain's first Muslim government minister, said: "It's a profoundly sad day for British politics and for our great democracy."
That the BNP won its first European seat under a Labour government will likely be seen as a humiliation for the party and in particular for Brown for failing to engage voters away from the far-right.
Although a reported plot by Labour backbenchers did not materialise last week, following a hasty cabinet reshuffle by Brown and declarations of loyalty by key ministers, there are fears the EU results could reignite a revolt.
At a meeting of Labour party activists in London on Sunday, Brown repeated his insistence that he would soldier on.
"What would they (the public) think of us if ever we walked away from them at a time of need... We are sticking with them," he said, adding that it was a "testing time for our whole country".
But after the partial results were announced, Labour MP John McDonnell said: "The message is clear -- we need a complete change of political direction."