Monthly Archives: October 2014

Studying the Test & Living the Predicted Life

There is a problem in our culture at this particular moment as it relates to mankind’s technological evolution. Data Mining has permeated most avenues of human activity; particularly those of readers of this post. The most common instances of it that we’d encounter are when we buy things online or watch something on Netflix. It’s also true of what we are likely to read — be it on Zite/Pulse or on the venerated New York Times — and what we see on the aisles of organised retail.

Of course it makes much sense and profit for those companies that’ve mined us; at least in the short term. The basic algorithm that is now about 20 years old is mind numbingly simple at its simplistic core: it merely discovers a rule for association with a certain level of confidence. That is, given you have bought X and Y you are likely to buy Z. Extend this for movies watched or articles read. The algorithms in those cases can be tweaked to something more fancy and tailored to one’s profile, but the recommendation problem is still Chapter 9 of this now classic textbook.

The problem as you can see is that the recommendation problem relies on a quasi closed set. That is, every subset of a frequent set has to necessarily be frequent itself. In other words, Z had to have been bought or watched or read enough times for it to be recommended to you. In the larger scheme of things, it simply means as a culture we’re now settling for exploit in the explore vs exploit problem that life presents us. Directors and producers who will invariably let this affect their art reduce the possibility of art transcending status-quo. It possibly also explains click baits on headlines and our reaction to it.

The other problem this creates is that it incentivises ghettoisation. For instance, one could argue, some 20 years ago the average person had a much higher likelihood of consuming news/commentary and general information from a wider variety of sources. Ever since this slide towards creating targeted content for specific groups started, with FOX News or Sun TV being good examples, the distinct constituencies have moved farther away from the original centre of public opinion. After all, once a bubble has been created it only makes sense for that bubble to feed itself. Its mean appears to be the true mean from within. And purists within will want to move the bubble ever farther away from their own and therefore the centre of public opinion. This process, as any of us who has a Twitter account can vouch for, has been exponentially accelerated in that part of media whose content is user generated.

The efficiency of tending to settle for exploit as opposed to explore makes sense for those selling. And makes absolutely none whatsoever for those buying. What we get is bad art, bad journalism, poor imagination in the design of products that’re sold and a far more intolerant society. A statistical understanding of the world is useful only when the world itself does not react to the statistical description of it. Since we’ve crossed that Rubicon, it’s reasonable to ask if Data Science as it applies to recommendation systems serves any positive purpose.

Imagining a Buddhist India

Recently, Dr De Weerdt mentioned one of the consequences of the Battle of Talas, in which the Chinese and Arabs fought each other for world domination in AD 751, was that the land routes for trade via central Asia were closed[1]. This according to her was one of the primary reasons for trade to shift to sea routes.

That theory fits quite well with the fact that the next three centuries were the glory days of Tamil civilisation. Simply by virtue of geography — sitting as it did in the middle of these traders from Arab regions and China — the Tamil country became extremely prosperous. It was also the period of Cholas at their height of their naval might and their fine bronzes. As Peter Francis notes in Asia’s Maritime Bead Trade: 300 B.C. to the Present, Tamil poetry from earlier and extending to this period discusses how the ships brought in gold to take pepper.

Remember this was also the period revival of Hinduism happened in much of what’s now India. As Dr De Weerdt noted, the plains of India were no longer the centre of Buddhist thought as they had been for few centuries until then. That must have meant there was already very little reason for Chinese and Japanese to come to India as they once did in large numbers resulting in spread of ideas. The closed land routes after AD 751 must’ve therefore isolated what’s North India for the same three centuries. Getting there was hard and there wasn’t much reason to. Japan and China were the new centres of Buddhist thought.

It’s amusing to think in the time of a Prime Minister who’s a Swayamsevak that the revival of Hinduism is possibly what caused the decline of India; and quite likely contributed to Muslim invasions a few centuries later. If Buddhism had reigned supreme, it’s conceivable and quite likely that the history of the world would have been very different. The Arabs wouldn’t have won the Battle of Talas at the least. And we’d all be speaking Mandarin, as well. And have had paper records instead of temple inscriptions from that period. Perhaps the Chola temples would then have not been as magnificent.

[1] – The discussion can be found here.

Should the INC Withdraw from 2014 Assembly Elections?

Assembly elections in the state of Maharashtra seem interesting and psephologically complex. The alliances have all broken down. A four cornered contest is both easy and difficult to predict; the leader should be reaping disproportionate returns but first past the post system in a large state with multiple distinct regions may complicate that. This post is not about polling data, however.

The election campaign in Maharashtra has drawn comparisons with the neighbouring state of Gujarat. It’s a bit of a strange comparison, given the former is a richer state with the luxury of having within its boundaries India’s premier city. It’s simply bound to do better in terms of revenues and investment. Historically, too, the state has been India’s industrial powerhouse. Therefore, it’s a bit meaningless to throw Maharashtra’s admittedly higher per capita income at Gujarat. Almost no one with even a peripheral understanding of India would dispute Maharashtra is richer in comparison to Gujarat.

It also makes no sense to credit Prithviraj Chavan, the incumbent Chief Minister, for Maharashtra’s better metrics in terms of the size of economy or its much higher FDI inflow. He did almost nothing to deserve it. Except perhaps in not ruining it. The only way to measure these two states on a relative basis then would be to understand how the state governments have performed on domains that are exclusively state subjects. There too, the base from which these two states started about a decade ago were different. Maharashtra has a heritage of higher literacy and better health. Gujarat doesn’t.

If we start with where these states were and how the relative performance has been in the recent past on health and education to see which government has done better, there’s not much difference between the two. We looked at this a certain way earlier; we’d measured how far current overall population illiteracy was from teen illiteracy. It showed us both Maharashtra and Gujarat were relatively poor performers, with Maharashtra having better absolute numbers because of its history.

In terms of health related indicators, however, we saw that Maharashtra is consistently in the top 5 states in terms of improvement. Not merely absolute numbers. That leaves Gujarat with worse absolute numbers and not even as good an improvement. So, if the choice were between being born in Gujarat and Maharashtra, one’d pick the latter without any hesitation at all strata of society.

But the above metrics are well known and therefore do not make for a non-trivial discussion. What does is this: should the INC, set for a certain defeat by most poll estimates, sit this election out for its own good? The senate race in Kansas is a possibly a good starting point for the INC to study. Greg Orman, an independent, is expected to win this reliably red state. Orman got on the ballot was by getting enough signatures to get on it. The Democrats noticing how well Orman was polling, dropped out of the race; the race thus became an Independent vs Republican race. Stunningly, the red state is now likely to send a possibly centrist Senator to Washington.

One could argue, the problems of the INC in many Indian states are remarkably similar to that of the Democrats in red state America. Its voters aren’t energised and its opponent the BJP is proving insurmountable in head to head contests. However, there are third and fourth forces in India already at play given it’s a multi party system. So, let’s assume the INC simply drops out of the race in Maharashtra. Will that mean a majority of its core votes go to a non-BJP formation that could emerge victorious? If that were indeed true, what’s the longer term risk of losing those votes forever to the other party vis-a-vis making a mere tactical retreat?

A cynical view of the AAP experiment was that it was intended to be the Greg Orman of Delhi. However the INC contested that election. Perhaps it did not have the courage to implement its own strategy. Just as the BJP could have done this when Sheila Dixit was on a roll but did not. When smaller parties contest every election as their survival depended on it, it’s understandable. They are likely to fade out of public consciousness if they don’t even contest elections. However, parties like the INC and BJP are big enough to not worry about that.

If they did that, it’d not only be in their self-interest – with a chance to stop the principal opponent from a certain victory – but also make Democracy stronger. Perhaps it’s time the major parties in each state came to an agreement: they’d not contest a third successive election if they’d won the previous two.

Swachh Bharat Abhiyaan: An Usurpation of States’ Rights

Many of us who watched Gus Van Sant’s Milk can’t ignore the parallel between Harvey Milk launching a campaign against dog poo on the streets of San Francisco and Narendra Modi’s recent campaign on Swachh Bharat Abhiyaan. Its politics is indisputable: everyone dislikes dirty streets, no reasonable person is going to oppose the idea and it gains media attention. The tested and proven formula of tagging others to increase the probability of the campaign’s virality was unfortunately not available to Harvey Milk, unlike to Modi.

The most important distinction between these two campaigns however is not the availability of experimental data on virality probability enhancing techniques demonstrated by the Ice Bucket Challenge; or that of a platform like Facebook. It’s that Harvey Milk was a member of the San Francisco Board of Supervisors. And at that level of government clean streets were his domain. So his campaign, however cynically exploitative, was still aimed at doing the job he was elected for.

Narendra Modi won the 2014 elections partly on a promise to send functions of the state which were usurped by the centre in the previous UPA government back to the state capitals. This was fair criticism; the rights expansion agenda of Manmohan Singh administration launched schemes such as the NRHM and Sarva Shiksha Abhiyan that trampled upon states’ rights and infringed on state subjects. In a federal structure the elected state governments are supposed to be responsible for health and education. The idea being that the more immediate a problem, the closer the seat of power addressing it ought to be.

One could try and make those arguments as to which domain and what authority need to be at the central/federal level and which ones need to be at the state level. From first principles. However, James Madison did that better some 225 years ago. Alexander Hamilton went a step further and wrote,

But it will not follow from this doctrine that acts of the large society which are not pursuant to its constitutional powers, but which are invasions of the residuary authorities of the smaller societies, will become the supreme law of the land. These will be merely acts of usurpation, and will deserve to be treated as such.

Not even Narendra Modi seems to dispute any of it if his own election speeches are taken to be evidence of his thought. And the most basic of such local government function is to clean out the street garbage. Admittedly, most Indian Cities do a bad job of this primary responsibility.

The reason for this failure of authority at the small society level could be many: perhaps local governments do not have enough powers or they may lack the political incentive or they may not have enough powers to raise revenue to pursue this basic function. There could be plenty other reasons which may require a study of its own and are beyond the scope of this post. But the question that this raises is: if small societies fail at their task, should the large society authority take over? And Hamilton, thankfully, has answered that already for us with a resounding NO. Which Modi agrees to as well.

The Prime Minister and his team may however put forward the following defence: this is a political messaging campaign to instil a change in mindset. That’s an argument that is, ironically, well supported by the fact that there is no actual policy or plan to achieve ends as far as Swachh Bharat Abhiyaan is concerned. It’s merely a call to action. Apart from being slightly bizarre in that the strength of an announcement is its lack of detail, this line of reasoning will be mildly insulting to people even if it were intended to have the philosophical rectitude as we assume it does.

The above line of reasoning essentially assumes local people as a group are incapable of achieving clean streets because they don’t know better. And it requires the national people/authority to tell them. Firstly, that’s politically insulting and Democracy has no place for such lecturing even if it were true. Especially in the absence of concrete funding. Secondly, we know that line of reasoning is empirically untrue now, given someone got a Nobel Prize for saying that, among other things.

The other line of argument that maybe put forward in favour of Modi is: this is political posturing and all politics is about posturing; at least this is something that can cause no harm. It can only make people act positively. But that is precisely the kind of optics that causes the dirt to swept under the carpet. The call to action shifts the blame on people instead of investigating if local governments don’t have enough powers of taxation or other forms of political authority. If the lacuna is that local governments lack the authority to raise resources they need to tackle this via enforcement before reaching the point of criticality to consistently ward off tragedy of commons, then that’s a serious problem that Modi has deliberately avoided. In the process, he’s also ignored addressing questions on federalism he touted were his differentiators. Worse, he’s exacerbated the problem by shifting the blame.