A useful way to think of organised crime is as the application of coercion for profit in social spaces where the power of the state does not effectively reach.
It is common to think of organised crime gangs as having “territories”. Such as this map of the territories of Mexican drug cartels.
Obviously, the drug cartels are not the only coercive power in such territory — the Mexican state still operates. Nevertheless, it make sense to talk of territories and turf wars — which can be vicious and deadly. So deadly, that law and order can break down if the struggle gets sufficiently intense. Ex-military personnel can be prime recruits for such gangs because they are trained in applying violence.
Researchers have found that American street gang territories in Los Angeles can be modelled by predator-prey equations, with violence concentrated on the boundaries of territories. (Abstract of article here.) Organised crime being a form of competition over scarce resources — income from crime; in the above cases, primarily from sale of illegal drugs. Though human trafficking is apparently rising in importance as an income source.
Essentially, organised crime gets its “in” because the state fails to maintain a sufficient monopoly of organised coercion. This can be because the state just does not bother (as in various shanty-towns and slums in Latin America or the Philippines). Or because its mediation and enforcement services are sufficiently incompetent, as happened in Bangalore in India, where organised crime provided land ownership enforcement and mediation services more effectively than the slow, complex and corrupt official land ownership registration, mediation and enforcement system. Or because the state bans a range of transactions, thereby withdrawing its mediation and enforcement services from them — given that said ban fails to eliminate such transactions.
Which is why command economies tend to become, over time, spectacularly pervaded by corruption. So many transactions people wish to engage in are illegal, that huge “black” markets form. Which then creates a demand for mediation and enforcement services that the state refuses to provide for such transactions; demand that is open (and likely) to be fulfilled by organised crime. Since command economies tend to have ruthless enforcement services that — by a thoroughly Darwinian process of selection — tends to breed ruthlessly effective organised crime (as they are the only ones that survive). A legacy that can persist (or even be exported) after the command economy has collapsed. Hence the notorious problems with Russian organised crime gangs.
The market for official discretion
Corruption is the market for official discretion, so the more pervaded by regulation based on official approvals the economic life of an economy is, the greater the level of corruption tends to be. In Australia, land markets have long been the major markets most pervaded by official discretions, so have a long history of corruption scandals. Since the Spanish and Portuguese exported highly economically controlling systems of government with elaborate licensing and other forms of official discretion, their former colonies tend to have notoriously high levels of corruption. Up until the mid C18th, English-cum-British politics were also notoriously corrupt. There was then a century or so of repeal of official monopolies, licenses and other regulatory interventions which massively shrank the scope of official discretions. By the mid C19th, British politics had a very high reputation for probity. Shrinking the realm of official discretions naturally shrinks the likelihood of corruption, just as expanding it tends to have the opposite effect.
Note that official discretions and regulation are not the same thing. One can regulate by some general rule or by requiring official approval; the latter is a form of official discretion, the former is not. The German Federal Constitution, for example, essentially blocks official discretions from interfering with (pdf) property rights. That effective ban does not stop Germany from having land and environmental regulation, it just forces it to be via open rules. The complication is that any regulation generates some form of official discretion in its enforcement; nevertheless, systems of explicit official discretion are much more vulnerable to corruption than rule-based systems.
While organised crime (in the sense of competing mediation and enforcement services to those provided by the state) and corruption are connected, they are not the same thing. In China, for example, the Communist Party elite dominates both the official state and the market for official corruption, greatly reducing the scope for any competing source of coercive services. Though whether this is a stable arrangement is another question, as continuing sale of official discretion is likely to undermine the social coherence, the asabiyya, needed to make such continuing exclusion of alternatives work. The Gu and Bo Xilai scandal may be the Party leadership drawing “lines in the sand” that organised violence will remain a Party-state monopoly.
Competing mediation and enforcement
Gambling, prostitution and drugs are notorious realms for organised crime precisely because they are banned transactions that the state has thereby withdrawn its mediation and enforcement services from which nevertheless continue to occur. As demand for such transactions are not eliminated by the ban nor such withdrawal, income sources are opened up able to support alternative mediation and enforcement services as neither party in such transactions wishes to attract the attention of the state. Prohibition was notoriously a hey-day for organised crime precisely because so many people still wished to consume alcohol. This connection between the wish to control other people’s transactions and willingness to supply banned services can lead to an effective alliance between the controlling moralists — the wowsers — and gangsters; the so-called “Baptists and Bootleggers” phenomenon. The notion that officially banning, or failing to recognise, something will make the thing go away is very attractive. Even if it is not fully believed, it can be a great way to express righteous malice towards others. A central appeal of political action is, after all, precisely that one can impose costs on others which one avoids oneself. (There is some depressing economic experimental evidence on the power of human malice.)
Protection rackets are the most direct form of competing mediation and enforcement services. (Anarchists of various stripes would argue, of course, that the state is just a protection racket that has managed to clothe itself in an aura of legitimacy.) The existence of protection rackets relies on people being either isolated from, or otherwise lacking confidence in, official state protection.
The failure of a state to enforce an effective monopoly of organised violence can have very unpleasant consequences. For example, there was a major drop in homicide rates in Western Europe after the medieval period (pdf) as the nascent modern state became the monopoly provider of organised coercion, rather than relying on that local franchising of protective services which we call “feudalism”. Coverage of protective services became less contested, so increasing in clarity and effectiveness. The state acquiring the income and administrative resources to be the monopoly provider being what led to the transition from “franchising” protective services to direct central provision. This led to a serious of inter-related “knock on” effects. As the nobility became less bellicose, its behaviour became more “courtly”, generating different social expectations. The expansion of centralised provision of protection led to less reliance on personal honour — a form of “privatised” protection — and less tolerance of its effects by state officers and judges.
Conversely, competition in the effective provision of mediation and enforcement, in organised violence, is competition in the ability to apply violence. An obvious way to demonstrate effectiveness is to eliminate the opposition, thereby reaping the benefits of local control. The recurring need to “prove” effectiveness, the dynamics of challenge and response, of displaying “honour” and protecting “standing” as per a system of privatised violence, and the problems of competing ambition both between and within gangs, can lead to high levels of violence. Whenever territory is the primary source of income subject to coercive capture, border violence is likely to be high. A major reason for the decline of warfare between states in modern times is that land is no longer such a central source of income. Conversely, conflict within states has been plausibly connected to high salience of natural resource income open to territorial capture (pdf).
Intentions and consequences
If one wants to reduce corruption, reduce official discretions and the extent of banned transactions. If one wants to reduce organised crime, ensure that the enforcement and mediation services of the state are effective and reliable — including well plugged into the local communities — and minimise banned transactions. Of course, the enforcement and mediation services of the state are more likely to be effective and reliable the less reliance there is on official discretions and the less banning of income-generating transactions.
But the desire to control the transactions of others — due to malice, righteousness, to generate or protect privilege, to provide income, to some combination of these — can be strong. And the consequences of high levels of official discretions or of banned transactions can be remarkably easily ignored or denied. Nor should we underestimate the willingness to inflict costs and misery on others for no other reason than preserve our own cognitive comfort. Particularly when all sorts of fears about what might happen if we don’t require official approvals, or ban said transactions, can so easily be conjured up and contrasted with the “manifestly” good intentions of such controls. Especially when we can associate ourselves with said good intentions and deny or ignore the consequences for others. A central appeal of political action being that one can impose costs on others which one avoids oneself.
Coercion is control over others without their consent. That reality is at the heart of organised crime but also of the policy and social failures that give rise to it.