LONDON At the height of an investigation into a group of Islamists plotting al Qaeda-inspired bomb attacks across Britain in 2004, British spies analyzed more than 4,000 telephone contacts to build up a picture of what they were planning and with whom.
The security services say the information was crucial in helping to thwart what could have been one of the deadliest attacks on Britain and to bring the cell to justice.
But a decade on, the police and intelligence agencies warn they have fallen behind those they are trying to track, as advances in technology and the growth of services like Skype and Facebook, increasingly put criminals beyond their reach.
In response, Britain is seeking to bring in what critics say are the West's most far-reaching surveillance laws that could change the international landscape in this area. The proposals would force communications firms to collect and store vast reams of data about almost every click of British online activity.
By doing so, ministers have provoked the wrath of human rights campaigners, sown division within the coalition government and alarmed major corporations such as Google and Microsoft.
"Nobody wants to live in a tyranny. I certainly don't and I don't want people snooping on what I do," said Gary Beautridge, the lead chief British police officer on the issue.
"This is about maintaining capability. It's not increasing capability, it's maintaining it in the face of change in technology," he said, rejecting talk of an Orwellian scheme.
Beautridge and all those involved in law enforcement say they are now unable to see about 25 percent of all communications data, hindering the secret war against bomb plotters, drug lords and pedophiles.
Almost everyone, from lawmakers to privacy campaigners, accepts something needs to be done. But trying to find a solution that is technically possible, will not cost billions and is not overly intrusive is proving a challenge.
Politicians across the world are grappling with the same problem but privacy campaigners say Britain is going further than any other democratic state.
Some countries such as France and Denmark are interested in new laws but campaigners said most other states had so far steered clear.
"The UK is the first mover in this. If the UK is successful, they will have changed the landscape for the rest of the world," said Gus Hosein, Executive Director of Privacy International.
Currently, if British authorities want to find out details about who has been talking to who they make a request to a senior police or intelligence officer who can approve the application without the need for a warrant.
British mobile and landline telephone providers must retain records for 12 months, in line with an EU directive, and figures for 2011 show some 494,078 applications were made.
Now Britain's Home Office, or interior ministry, plans to expand these powers to include online activities, such as which web sites were looked at and who was talking to who on social networks.
The new law would force British Internet service providers (ISP) and mobile operators to store data they would not normally keep for billing purposes and could even require them to keep data generated by internet groups based outside Britain.
The government insists it does not want to look at the content of the exchanges, but merely the details of the contact.
"NO FISHING EXPEDITIONS"
"All we're talking about is keeping the communications log of actually who owns that account, when a call was made and to whom or from whom and that is it," Beautridge said.
"It's not the content, it's never the content. Fishing expeditions should not happen. The processes have been designed to ensure that."
However, last December a draft version of the bill was heavily criticized by a parliamentary committee, which said it was too sweeping in its remit, would give ministers too much power and was likely to be too expensive.
The government is now expected to outline an amended bill in weeks with many of the same powers, according to sources familiar with discussions, having made it clear it cannot wait.
"Technical experts are clear that everything in the bill is feasible and we are continuing to consult with communications service providers on our proposals," said a Home Office spokeswoman, adding the aim was to bring a law forward "at the earliest opportunity" along with its cost implications.
Unlike the original plans which placed huge powers in the hands of the home secretary, the new proposals are likely to give parliament a greater say in what is permissible.
But privacy campaigners warn that might just put a gloss on something still unpalatable, putting in place a system which could be extended at a later date.
"If the police want to investigate me, they should be able to ask any company that has data on me to disclose that information. That's generally not a problem," said Hosein from Privacy International.
"The problem is the Home Office want much more than that. They want these companies to record these activities just in case at some point in the future I may become a suspect. That's not the way things work in a democratic society."
Among the concerns are proposals that service providers keep weblogs - records of websites people have visited - and a "filter" system which would ask them complex questions and filter data accordingly, something Hosein describes as mass surveillance.
"Having a list for 12 months of every single website you go to, that's quite a lot of information about yourself, very personal information," said Julien Huppert, who speaks on the issue for the Liberal Democrats, the junior members of Britain's coalition government.
But it's not just the rights or wrongs of the system at stake, there are also concerns it will not work in a world of mass communications provided by firms located across the globe.
One of the main sticking points will be how the authorities get information from so-called third party service providers based outside British jurisdiction, such as Google's Gmail, Facebook and Microsoft's Skype.
British-based mobile operators have told Reuters they are happy to cooperate with the government, but they insist that the same rules must apply to the likes of Facebook.
"From a security point of view, you need to be able to have access to the full pool of communication otherwise you're fishing in a sub-set of a sub-set," said Ronan Dunne, the chief executive of O2 UK.
If internet groups based outside Britain do not comply, the Home Office envisages forcing the British Internet Service Providers who carry their services to access the data instead, through a process known as deep-packet inspection.
But, it is not clear if this will be technically possible. Google has said it would not allow another service provider to decrypt its information on its Gmail service, and Jimmy Wales, the founder of Wikipedia, has said he would not cooperate.
"If we find that UK ISPs are mandated to keep track of every web page that someone reads on Wikipedia, I am almost certain ... that we would immediately move to a default of encrypting all the connections to the UK," Wales said.
The government, which has already spent 400 million pounds so far on the scheme without it getting off the ground, has estimated that it would cost 1.8 billion pounds over the 10 years until 2020-1, a figure that has been disputed. The Treasury has also said the funds have not been approved.
"We got widely divergent estimates of the cost and none of them were lower than the Home Office suggestion," said lawmaker Stephen Mosley, a member of Prime Minister David Cameron's ruling Conservative Party who sat on the parliamentary committee which scrutinized the draft bill.
Polls indicate that the majority of Britons do not like the plans, think they are a waste of money and do not have faith that the information will be secure.
Mosley, co-chairman of the Parliamentary Internet, Communications and Technology Forum, warned it was vital that any law could not give succor to draconian regimes which sought to censor their citizens activities.
"We as a country do want to be a beacon for freedom and liberty. We've got to make sure that legislation we introduce can't be misused elsewhere," he said.
(Editing by Guy Faulconbridge and Anna Willard)