LONDON (Reuters) - Blindsided by the "Arab Spring", taken aback by the speed of the economic and geopolitical shift to Asia and apparently always a step behind on the Eurozone, Western governments are putting new effort into "horizon scanning" for coming seismic global shifts.
For the United States and its allies, some worry the heightened focus on Iraq, Afghanistan and the "war on terror" meant the bigger picture was far too often ignored.
While no one expects truly accurate prediction, senior officials and managers complain that the last half-decade has too often seen specialists and experts failing to notice dramatic change on a scale not seen since the - equally unexpected - fall of the Soviet Union.
Britain's Cabinet Office - which houses the analysts who pull together intelligence assessments for the Prime Minister and other senior officials - is conducting a review of its "horizon scanning" and ability to think about long-term trends.
Other governments and major institutions are often taking similar steps.
"What I need from horizon scanning is not a focus on everything that might happen in the next 6 to 12 months," Jon Day, chairman of Britain's Joint Intelligence Committee, told an audience at Foreign Office-funded conference centre Wilton Park outside London on Tuesday. "What I need is more advanced warning of the really important paradigm shifts."
He was speaking as part of a conference-cum-course on "international futures" attended by senior policy planners from a range of governments and companies.
Some officials working in policy planning units already proudly describe their teams as being "on the edge of chaos". They say they welcome "wacky thoughts" and use the widest available range of sources, strategies and techniques to stay ahead of the curve - or at least to avoid falling too far behind it.
But most, both officials and outside experts suspect, have tendencies towards conservatism and secrecy that can make their task even harder. Staff theoretically responsible for looking at long-term issues often find themselves pulled onto working on the crisis of the day.
Even when organizations such as the CIA, US National Intelligence Council or Britain's Ministry of Defence produce major reports looking at long-term future risks, policy insiders say they are often largely ignored in the day-to-day rush.
"The global economy is going through a period of transformation," says Mansoor Dailami, manager of the emerging global trends team at the World Bank in Washington DC. "But it is sometimes difficult to appreciate history when you are living through it."
For cash-strapped governments, hiring expensive new teams is often out of the question and the focus is on getting more from existing resources. But the most experienced analysts and experts can also be amongst the hardest to convince that something unexpected might be brewing on their patch.
"It's about challenging assumptions," says Michael Oppenheimer, a professor at New York University who works on scenario planning with the U.S. government and intelligence agencies. "For most experts on most countries, 'muddling through' becomes the default scenario. They have a tendency to underestimate the likelihood of profound change."
For government institutions often designed to handle the most secret intelligence, some of the skills needed to keep ahead of trends in an increasingly interconnected, modern world may be tough to take on.
Secret discussions with other governments, reports from spies or intercepted communications can still sometimes prove vital. But more often in the 21st century, the real challenge can be to tackle the vast volume of information already available on the Internet and elsewhere.
The lesson of the revolutions in Egypt and Tunisia in particular, some current and former officials with knowledge of secret intelligence say, is that the governments of those countries themselves had little clue of what was about to overwhelm them. Getting a truly accurate picture of current events and what is to come, they say, increasingly means looking well beyond the standard sources and contacts.
In a world more connected - if not necessarily more complicated - than ever before, some planners study chaos or complexity theory. Models imported from the private sector - such as the scenario modeling systems used for decades by oil giant Shell - are also increasingly being carried across into government.
The aim of such techniques, supporters say, is not to necessarily identify the final trigger for an event such as the "Arab Spring" - in that case, the self-immolation of a Tunisian vegetable seller.
But simply by thinking through various possible outcomes, those involved hope they develop new insights and may be better prepared for the surprises that actually come.
"You do not expect... to necessarily take action ahead of an event that, say, has a 30 percent probability," says New York University's Oppenheimer. "What you hope is that... (you) become more aware of these are possibilities."
But the process of scenario planning, done properly, can be a bruising experience very different to the more staid discussions bureaucrats were used to.
"Every one of these (scenario planning) meetings has a moment where you think everything is going to fly apart," Oppenheimer says. "People get angry. People disagree."
Whether such meetings genuinely impact on government activity, however, can be almost impossible to prove. Even supporters of horizon scanning and sophisticated scenarios work concede that all too often it can simply be ignored, set to one side by policymakers already overwhelmed.
Some of the larger trends - such as the shift from a U.S.-dominated unipolar world to a much more much more multipolar system with several great powers - can be more than a little intimidating to some.
In Washington DC, some officials and pundits worry there is little honest discussion of how the United States should position itself, with the Obama administration concerned about being accused of embracing the idea of "American decline".
Chinese officials occasionally quietly worry that Beijing too lacks the requisite skills and systems to manage an uncertain future, and a rapid emergence on the world stage may bring with it multiple opportunities for strategic blunders.
Politicians and senior officials also complain that those sketching out the scale of potential future change - whether climatic, economic or strategic - fail to appreciate the political and financial constraints of the real world.
Perhaps the greatest lesson of recent years, some believe, is that governments need to be much more aware of what it is they do not know.
"We're simply not good at predicting the future," one Western official involved in policy planning told Reuters earlier this year. "What is important is remembering that."
Reporting By Peter Apps; Editing by Peter Graff