Can governments be more innovative than private enterprise?

I have had some interesting thoughts lately regarding the trends towards the privatisation of infrastructure and the user pays principle which underpins this trend. My theory suggests that private infrastructure based on user pays principles locks society into a particular path of development which becomes increasingly self-reinforcing, thus excluding innovative solutions to transport, communications, energy and water supply.

Let me explain in more detail.

Consider two countries, A and B. Previous governments of country A have spent the past century investing heavily in a rail network for both passengers and freight, while country B has spent the past century ignoring rail and building roads as the major land transport system.

Now imagine that a technology, X, is developed that can massively increase the efficiency of rail transport, but not road transport. Think along the lines of mag-lev trains or some such thing.

Now both country A and B believe that this technology is superior to their current land transport system and aim to develop a network based on private investment. Country A already has the land, the stations, and the infrastructure in place, while country B has none of it. One would expect that the compared with country B, country A is more likely to find potential private investment for such a project given the likely lower costs but equal benefits.

Thus due to historical capital investment, country A continues along a rail based path towards the most efficient outcome available with technology X, while country B continues along a road based path and will never reach technology X through private investment alone and will remain 'stuck' with a less efficient land transport network. This is the problem of path dependence, a situation encountered in both evolutionary theory and economics –“the cheapest manufacturing method may not be achievable by “evolutionary steps” but may require a complete change in method”.

The question then arises that if country B is ‘stuck’ on a more costly trajectory of land transport development, how does it become ‘unstuck’. This is where I believe governments can intervene to make the decision to become unstuck by directing investment into the superior new infrastructure. Rather than the user pays principle of privatisation, the government can justify funding such scheme due to benefits to the user, but also benefits to non-users in society.

For example, if a rail line is established along a popular road corridor, both the rail users and the road users benefit - the rail users from cheaper transportation, and the road users from less congestion. Unless a private enterprise can charge the road users for the rail line, a publicly funded outcome is far superior.

The inability for private infrastructure owners to capture external benefits limits their opportunity to innovate and provide broad social benefits.

Governments on the other hand can consider all external benefits and consciously ‘invest’ a region out of their current development trajectory on the basis of providing indirect social benefits.

The problem then is getting a government to even perceive these potential social benefits, let alone act on them.

The other problem is that if governments do ‘invest’ a region out a of a particular development path, they are now stuck in the new path, unless they invest heavily once again. For example, Brisbane ditched trams back in the 1960s, and successive governments ever since have been considering getting them back. But each change in trajectory is more expensive than the last.

So what then should a government do to maximise social welfare? Should they just stick to the path they are on and hope that future technologies are advantageous, or should they embrace innovation and change the development path?