Doesn’t India Owe Reparations to Dalits?

Two individuals have argued for reparations in two very different contexts in the recent past. There was Ta-Nehisi Coates making an arresting case on reparations to African Americans for slavery. Then there was a slightly more gimmicky Sashi Tharoor making the case that Britain owed India reparations for colonialism.

Coates’ basic premise is indisputable. America is what it is today because it was built on slavery. And for 350 of the 400 years its existence, African Americans have been targeted and exploited and not let to realize their potential in a systematic and state sanctioned manner. That as late as 1960s the rules of federal and local governments were such that African Americans could not really get to own property takes a while to digest. If a community does not own property, of course its social capital is going to be so limited. And then add to it all the other forms of continuing discrimination.

Coates’ second book, Between the World and Me, is the kind of polemic that one wishes a Dalit writer in India wrote. Viewed through this prism, Britain owing India reparations seems insignificant both as a moral and economic debt compared to what India owes its own Dalits. The African American, after all, had been enslaved only since 1619. The Dalit has been denied property rights and consequently generational growth for millenia. The social ostracism’s cost on culturally influenced personality aspects, such as perseverance, may be even higher.

Repairations to Dalits raises an important question on the construct of civilization. In recent times, a lot of liberalism has been explained by outcome. For example, having a diverse classroom that includes Dalits we are told improves the overall learning outcome. That may well be true. But what if it isn’t? What if paying that moral debt is not additive in civilization’s continuous arc of greater consumption? What if liberalism is not utilitarian? Coates never gives up on that arc of ever greater consumption.

A Dalit writer approaching this topic may well give that assumption up. I wish to read that.

Let them farm

Tim Gowers recently lectured on the possibility and reality of machines proving theorems. Higher mathematics is one of the human endeavors that we think defines us as a species and hold it dear. Increasingly though it appears we may not be as special as we thought we were.

While Gowers stands at the edge of human thought, the real impact of machines on our political economy was detailed by Andrew McAfee in his book Race Against the Machine: How the Digital Revolution is Accelerating Innovation, Driving Productivity, and Irreversibly Transforming Employment and the Economy. McAfee’s basic point is: this time it’s different. And that unlike all technological progress of the past which resulted in greater prosperity and more jobs eventually, the coming automation will actually kill jobs. An odd thing for an Economist to claim. He even proposes the masses rendered unemployed thus be paid salary for doing nothing. His arguments, whether or not true for mankind, have a very important bearing on India

India’s problems, Mihir Sharma’s analyses in Restart: The Last Chance for the Indian Economy suggest, stem from factor markets and bad laws governing them. Sharma then goes on to suggest reform in these areas to address said problems. That’s accurate if the next 30 years are similar to the past 30 years. But what if the option that was available to China is no longer available? Whether there can be another export/manufacturing driven large country is a question for Economists. But a simpler question that the rest of us have is: the imperative for moving manufacturing to low wage destinations becomes moot if humans are no longer needed to perform semi-skilled or even skilled factory jobs, does it not? What then? How does labor reform matter when productivity doesn’t depend on people?

Even Computer Scientists can’t predict what will be automated in economically viable ways in the near or medium term. Politicians stand no chance. So to invest in skill development programs that train people for semi-skilled jobs when those jobs may well be done by a machine in the next 10 years seems a worse option than subsidy for fossil fuel.

What does a politician  in India then do? Tell the populace its strength is the over-educated upper caste/class which will at least participate in the global economy unlike in some other countries? And that the lower caste/class farmer in an unproductive piece of land with almost no access to education might as well stay that way? What we can say with reasonable certainty is: elections are going to be difficult and anti-incumbency is likely to be an explanation for many future election cycles in India.

How did Tamil Nadu get richer than Gujarat?

In their recent book The Paradox of India’s North–South Divide, Samuel Paul and Kala Sridhar discuss how and why Tamil Nadu and Uttar Pradesh have diverged quite significantly over the past 30 years. While their aim was to establish factors that primarily improve per capita incomes of the southern states, the book is limited to those two states largely. And was almost asking the reader to do a wider analysis for oneself. So, let’s look at a wider set of states. Let’s also include what one thought was an important factor that was missed in the analysis: population growth of each state.

Let’s consider 16 large states and their per capita incomes in 1961 and 2012[1].

State Per Capita Income (1960) Relative Rank in 1960 Per Capita Income (2012) Relative Rank in 2012 Movement in Relative Ranking
Kerala 278 10 88527 5 +5
Haryana 359 5 119158 1 +4
Tamil Nadu 344 7 98628 3 +4
Orissa 226 15 49241 12 +3
Jammu & Kashmir 267 13 52250 11 +2
Rajasthan 271 12 59097 10 +2
Andhra Pradesh 314 8 78952 7 +1
Karnataka (Mysore) 292 9 76758 8 +1
Bihar 216 16 27202 16 0
Gujarat 380 4 96976 4 0
Maharashtra 419 1 103991 2 -1
Uttar Pradesh 244 14 33616 15 -1
Madhya Pradesh 274 11 44989 13 -2
Punjab 383 3 84526 6 -3
West Bengal 386 2 61352 9 -7
Assam 349 6 42036 14 -8

50 years is a reasonable time frame. And the data is quite remarkable. Kerala and Tamil Nadu had per capita incomes that were almost the same as that of Madhya Pradesh and Assam respectively in 1960. In 2012, Kerala had a per capita income that’s about twice that of Madhya Pradesh; Tamil Nadu had a per capita income that was well over twice that of Assam in 2012. Maharashtra and Gujarat have held on to their relative wealth. Andhra Pradesh (including Telangana for the purpose of this analysis) and Karnataka continue to be stuck in the middle. While West Bengal, apart from Assam, has seen a collapse in its relative wealth; Punjab, still wealthy, has lost its position a bit.

The relative movement, interesting in terms of challenging our political conceptions, does not tell us much. A far more interesting comparison is to look at growth rates of per capita income and compare it to actual NSDP growth.

State Per Capita Income CAGR NSDP CAGR Population Growth Rate(1961 – 2011)
Haryana 11.81% 14.76% 1.40%
Kerala 11.72% 13.58% 0.99%
Tamil Nadu 11.50% 13.25% 1.07%
Karnataka (Mysore) 11.31% 13.62% 1.23%
Gujarat 11.25% 13.89% 1.32%
Andhra Pradesh 11.21% 12.06% 1.15%
Maharashtra 11.19% 13.82% 1.30%
Punjab 10.94% 13.36% 1.20%
Rajasthan 10.91% 13.85% 1.41%
Orissa 10.91% 13.26% 1.16%
Jammu & Kashmir 10.68% 13.71% 1.43%
Madhya Pradesh 10.31% 12.34% 1.34%
West Bengal 10.24% 12.54% 1.24%
Uttar Pradesh 9.94% 12.37% 1.26%
Bihar 9.75% 11.71% 1.32%
Assam 9.65% 11.90% 1.24%

The two data tables taken together for the states of Tamil Nadu and Gujarat tell a very important story: while Gujarat’s NSDP was growing slightly faster than Tamil Nadu’s, the latter had a faster growth of per capita income. Simply because it had a lower rate of population growth. So, in 1960, Gujarat had a higher per capita income compared to Tamil Nadu. And in absolute terms, its economy grew slightly faster than Tamil Nadu in the past 50 years. Yet, at the end of those 50 years, the average Tamil is richer by about 1,652 rupees.

Kerala and Tamil Nadu, thus, aren’t growing faster than others. Just that they are growing just as well as others with fewer babies. Tamil Nadu has a TFR of 1.7 and Kerala’s is 1.8; Gujarat’s TFR was 2.5 in 2010. This story repeats itself with a far greater growth dampening effect in the case of Madhya Pradesh and Bihar. Both states had TFRs greater than 3 in 2010.

On the other hand, fewer babies isn’t a sufficient condition for doing well. West Bengal with a TFR of 1.7 sadly explains that. If we treat per capita income as an independent variable, the relative weights between NSDP growth and population growth is still skewed towards the former as shown the chart[2]. But among relatively well governed states, the difference in growth is negligible while that in TFR isn’t.


The other factor that’s implied by Paul and Sridhar is that States with a greater share of services in their NSDP tend to break out compared to states that have a higher contribution of agriculture or manufacturing. This combined with low fertility is likely to open the advantage of states with high literacy and low TFR even further. Whether this adds a cleavage towards secession in the future will be worth watching.


[1] – The data for 1961 per capita income, NSDP figures was from these statistical tables. The data for 2012 NSDP Figures were found here; and the Per Capita Income was from Planning Commission. None of the figures have been adjusted for inflation. I assume because they are compared and relative movement is what’s used for analyses, that’s reasonable. If you think otherwise, please let me know. Madhya Pradesh, Bihar and Andhra Pradesh have been treated as undivided states. I calculated Population Growth Rates using Census Data of 1961 and 2011.

[2] – The details of regression and relative weights from R are as follows,


Silver Bullet: Fighting Crime against Women

Crimes against women varies over a large range across the various states in India in terms of their rate of occurrence. This may have multiple reasons but it may also be complicated by the fact that poor law enforcement in poorly governed states may result in under-reporting of crime. But if we assume the NCRB data to be true, the distribution across various states of India is,


The states’ ranking in this is odd when one uses the prism of conventional wisdom in terms of advanced and backwards states.

A natural place one will want to look at next is police presence. Some Indian states have truly bad Police-Population ratios. Perhaps that’s a contributing factor. Another factor is number of police stations — perhaps a greater distribution of police stations as opposed to actual number of police personnel is a factor. An analysis of their scatter plots suggests these factors’ relationship with crime against women isn’t as strong as one’d expect them to be. Their respective scatter plots (with a regression line in red) are,

Police Station







The overall police presence as a ratio of population does seem to have some negative effect on crimes against women compared to police stations’ spread. But it’s not as strong as one’d expect.

Now consider the other factor: number women per police station in each of these states and female workforce participation in each state. Their respective scatter plots with a regression are,







These two factors, namely the presence of women in police stations and their participation rate in the workforce, seem to have a significant negative impact on the crime against women. The first factor is one of possibly bringing a female perspective to policing improving security. The other is that women who work outside their homes seem to actually reduce the likelihood of crimes against women.

A simple way to compare all of these factors will be to have a regression model that includes all the factors and measure each factor’s relative weight in the model. When one does that, the relative weights show up as,



Thus, the domineering factor as we see is presence of women in police stations. States that have more women in each station do better on preventing crime against women even if they have a worse overall police to population ratio. Consider two relatively advanced states for an example: Kerala and Maharashtra. While Maharashtra has fewer police personnel per every lakh of citizenry compared to Kerala, it has far more women as a proportion of it. The state of Maharashtra also records fewer crimes against women.

Of course none of this can be considered causal and under-reporting in Bihar and Uttar Pradesh are likely to undo the above analysis. But a simple but effective strategy for states in fighting crime against women appears to be recruiting more women in their police force. And having more financially empowered women in the population.

[1] – Data on crimes against women was from NCRB

[2] – Data on Police Organizations was from BPRD

[3] – I have collated the data in a spreadsheet. Should you want it, please find it safety

Life in the time of data

With very few exceptions, Data Science and Machine Learning as they exist today are about optimizing processes. We mostly predict likelihood or target resources better or categorize things. Sometimes using fancy algorithms and at others using basic generalized linear models. But in general we expect the cartoon of our past to predict our future.

There is plenty of criticism this data driven culture is subjected to. Much of it comes from uninformed people in social sciences. Their objections are often trivial. A far more important and troubling objection is: this reduces civilization to actually optimize status-quo and not advance. The most famous example of such refutation is Kepler and his laws of motion. That was uniquely a human insight that no algorithm can arrive at; even today. And it was essentially insight from data. Almost no one will argue they can design a system, with how many ever clever back propagation networks, to arrive at what Kepler did from the data he had. No one’d even claim we’d reach a stage where algorithms can do that any time soon in the future.

The point of that example is: do we no longer contemplate about data sets as deeply as Kepler might have simply because our belief in algorithmic processing of data is so strong that we see no merit in holding the entire data set in our heads? We possibly see that as data equivalent to rote learning; a waste of critical brain resources that can be freed for doing higher things.

A related aspect in our modern lives is the absence of intellectual silence. We’re often surrounded by things and people often far more impressive than ourselves. It almost sounds rude to think one’d think for oneself when the finest of the world on that exact topic is ready to speak to you, albeit in a lecture or podcast, in the device you hold in your pocket. It’s unlikely that one is going to out-think what’s available in one’s pocket. But what seems the honest thing to do in that instance certainly makes our overall epistemology suffer a lack of originality. It’s easy to imagine a world where originality of thought is rare and we’d all be poorer because of it.

A useful technique then is to possibly alternate between these two states by consciously allocating their specific times. But any such a-priori allocation of time for categorizing thought sounds awfully arrogant in itself. We could possibly model ourselves on, say, Kepler. And apportion our consumption of thought and thinking accordingly. Which defeats its own very idea. We certainly live in difficult times.

Fourteenth Finance Commission: Has Federalism Arrived?

The Fourteenth Finance Commission’s(FFC) recommendations and the subsequent spin on it by both the Prime Minister and editorialised news reports have made it sound as if federalism in India has finally taken off. Is that the case though?

To begin with, it’s useful to understand what the original demand was. The AIADMK, for instance, made that part of its election manifesto: a constitutional amendment so that the revenue raised from the states through cess etc should be equally shared with the state in question. A radically different and truly “federal” request.

What the FFC did  instead was raise the overall share of all states as a percentage. The FFC’s discussion on this even says they settled for it since an amendment did not look possible. FFC89The second option that the FFC has chosen transfers more money than previously to all states combined; but that was not the original demand. This comes no where near the demand of: the state where the collection of taxes happens should get its share based on that collection straight. It’s understandable why Dr YV Reddy decided to to opt for what he did: after all, the commission he headed has no powers to amend the constitution. But what is a blunt increase in the volume of money pumped out is somehow sought to be passed off as a a paradigm shift in federalism which it surely isn’t. This system is still one where one partner gets the revenue generated in the others’ domain and then decides how much the other should even get of that. The other aspect is how this devolution takes place and how the FFC arrived at the relative weights of the various parameters. The end result is, weightsffc   And remember the factors of Demographic Change and Forest Cover have been introduced in the FFC. Both these factor are a significant cost to states like Tamil Nadu,

  • which have low fertility rates and therefore their demographic change is among the lowest[1]
  • have low forest cover because of their geography and because the state has extremely high degrees of urbanisation

It’s interesting to see what the states themselves sought these weights to be. As one’d expect, Tamil Nadu, West Bengal and Maharashtra which were populous states historically but have shown a remarkable decline in fertility rates since, want the population of 1971 to be accorded maximum weight and that of 2011 nothing at all. Bihar, Madhya Pradesh and Uttar Pradesh want 2011 population to be accorded maximum weight and none at all to the base year of 1971 as the terms of reference for Finance Commission states.popuffc


The FFC has a discussion on the various requests put forward by the states. But it does not seem to explicitly state how it arrived at the final weights. And these weights seem to be a simple middle path between Tamil Nadu and Uttar Pradesh; at least that’s what it appears to be to the untrained eye. An odd end point for a group of gifted Economists.

The result of all this is, taxes collected in Maharashtra, Tamil Nadu, and Haryana will end up funding BIMARU states. One could argue this is progressive taxation. In that it aims to get from rich states and give it to poor ones. There may or may not be merit to that form of progressive taxation, but it certainly can’t be called ‘paradigm shift in federalism’. Federalism in essence is what happens when the unit of political discourse is the state, not the Union of India. And this FFC explicitly sticks with its unit of discourse as India.

சாதியும் நானும்: Personal Essays on Caste

Perumal Murugan has been in the news recently for tragic reasons. He’s since moved jobs and cities. Analyses of the author’s struggle has been, justifiably, about the politics and not his literary output[1]. In the lead up to Murugan declaring himself dead and living in exile, he answered a few questions. The Streisand effect of this entire episode did make many pick a Tamil book for the first time in their adult lives. One that I did was a collection of personal essays that Perumal Murugan had edited, titled சாதியும் நானும் (Caste and I).

The essays are written mostly by former students and colleagues of Murugan, who now teach Tamil in various colleges across Tamil Nadu. They largely hail from the many middle and lower castes of central/western parts of the state. Each personal essay is short — only a few pages long — and often narrates a single or set of anecdotes. By their very definition, all of them are written in first person singular. They take us into the worlds of their castes and caste rituals that are strange even for those of us living in the state and think we know it reasonably well.

There are women who’re forced to bathe naked in the middle of the night in open air, men who’re forced to eat from the floor straight and students made to work in farms without pay. The cruelty of some practices and how they aim to dehumanize the person of lower caste on a consistent basis is an ugly window into mankind’s worst. And this caste hegemony, observed from such close proximity, isn’t merely aimed at other/lower castes; its notions of purity to mask the underlying misogyny against women of the same caste is heart breaking.

The essays themselves, however, are quite limited in their scope. They do not attempt to fork in the requisite distance between essayist and subject at any point in time. There isn’t a narrative arc even, to most essays. On the rare occasion some author attempts that, it ends up being a polemic that doesn’t seem honest. There’s a lack of an exploration of one’s own human condition; however pretentious that goal is, its absence appears far worse an alternative. The frustrating part is, one tends to agree with the polemic. And wishes the essays were written with greater emphasis on the craft of writing itself.

Perhaps this was what Perumal Murugan aimed to achieve and therefore forced the structure of a limited personal essay on the various authors. And first time readers of the personal essay in Tamil miss the whole point of it.

[1] – This gentle introduction to Perumal Murugan was an exception.

Are Indian Muslims Outbreeding Hindus?

The narrative that Muslims have too many children and are out to outbreed Hindus isn’t merely a fringe notion held by Sakshi Maharaj or the VHP[1]. The incumbent Prime Minister held that view as Chief Minister of a large state and hasn’t mentioned a revision in his views since[2]. It’s reasonable to assume therefore, the Hindu right’s position more or less is: each Muslim woman has at least 4 children as part of a demographic conspiracy. Is that true, though?

According to 2011 Census, 14.2% of India’s population was Muslim[3]. Up from 13.4% in 2001. The growth rate of the Muslim population compared to Hindu population has significantly declined in the last 2 decades but it still is growing[4]. Is that evidence for the Hindu right’s assertion?

There are a few ways to verify this. The first is of course to do a simple scatter plot of fertility rate of various states against their proportion of Muslim population; and draw a regression line through it[5]. What happens if we do that is a near flat line — not a strong relationship at all.MVF

The correlation coefficient between Muslims as a proportion of population and fertility rate is 0.14. Low enough to ignore it. Especially when you consider what the relationship between female illiteracy of the states and their fertility rate is.

ILFThat’s a much more powerful relationship without even looking at summary statistics. The correlation coefficient between female illiteracy and fertility is 0.67. Quite a strong correlation.

In case the Hindu right wants to make a convoluted argument that there’s possibly a second order effect, that is Muslims cause low female illiteracy which leads to high fertility, they’d be out of luck. There is negative correlation (-0.17) between female illiteracy and Muslim population. In other words, you’re more likely to see higher higher literacy in states with more Muslims.

There is also a simple back of the envelope calculation that supports the fact that Muslim women aren’t out to produce 4 children each. Consider the large states that have highest percentage of muslim population: West Bengal (27%) and Kerala(26.6%). Their fertility rates are 1.7 and 1.8 respectively. If we extend Narendra Modi’s metric of each Muslim woman producing 4 children to either of these states, their Hindu fertility rate[6] would fall 0.84 and 0.99 respectively. That’s so obscenely low that it means Hindus are about to be reduced to half their current size or even less in a generation. For comparative scale Japan, that famously ageing and depopulating country, has a fertility rate of 1.41. Surely no one, not even VHP, thinks Hindus in Bengal are almost twice as worse compared to Japanese in terms of depopulating.

If the data above points to Muslims not outbreeding Hindus, why does their population as a percentage keep growing in comparison to Hindus? That possibly has a simpler explanation: Uttar Pradesh and Bihar. These two states are India’s most populous and have above average representation of Muslim population; 19.3% and 16.9% respectively. Their fertility rates, as states, happen to be 2.6 and 3.6. Above the fertility rate of 2.4 for India; and well above that of peninsular India at 1.8. South India and as a whole has less number of Muslims as a percentage of population and lower fertility rates as a whole. If we adjust for above average Muslim presence and above average fertility rates in UP, Bihar and Jharkhand (14.5% Muslims and 2.9 TFR) what we will get is the near flat regression line of the first graph. The summary statistics of that is,

Muslim_Population MuslimsFertility

What’s disappointing is that the literature is dominated by Sociologists who take for granted Muslim women have higher fertility rate and then try to explain it[7]. Without a simple back of the envelope calculation that’d suggest the assumption is not even valid.



[1] – Both Sakshi Maharaj and VHP seek 4 children per Hindu woman.

[2] – Modi’s speech on Hum paanch, humare pachees

[3] – Data from Census

[4] – Changes in Fertility Rates Among Muslims in India, Pakistan, and Bangladesh

[5] – Data from Census for percentage of Muslims and SRS for state-wise fertility rate. Find the CSV of combined data here.

[6] – For the purpose of this calculation, non-Muslim population is assumed Hindu.

[7] – Muslim attitude towards family planning

[8] – Social Development and Demographic Changes in South India: Focus on Kerala


Should the Vote be Sacrosanct in a Democracy?

North Carolina had a reasonably close race by the standards of mid-term elections in 2014. Thom Tillis, the Republican challenger, defeated incumbent Senator Kay Hagan by 1.7 points. Or, looked at the other way, by 45,511 votes. The Libertarian Sean Haugh polled in 108,183 votes[1]. That is, Hagan’s margin of defeat is roughly half of what a fringe third party candidate polled — a good measure of how closely contested the election was. Six years ago when Hagan defeated the then incumbent Elizabeth Dole, she’d polled 2.24 Million votes and won by some 8.5 points or 361,781 votes.

What’s interesting in the two elections is obviously the fact that in 2014 a vote seems to have had a lot more impact than it did in 2008. There were 6,629,703 registered voters in North Carolina this time. If all of them had voted, each voter would have had an impact of about e^(-1358). That’s absurdly low even when compared to what an Indian citizen has in terms of electing an MP. But these are Senate races, and are state-wide[2].

But the difference is that instead of the over 6.62 Million votes only 2.88 Million were cast in 2014. In 2008, 4.27 Million were cast. That puts the impact of each voter in 2008 at e^(-1032) while in 2014 it jumped up to e^(-847). In the absurdly low stakes, that’s still a many-fold jump from 2008 to 2014. By orders of magnitude. So what’s clearly in favor of the individual voter is to not have many voters turn up but the voter turns out herself.

This intuitive and mathematically obvious truth leads us to a question: is one vote per person in all elections the most optimal way to structure a Democracy? After all, some elections may be about nothing; as the recent mid-term elections were touted to be. Others maybe about pot or fiscal policy or foreign policy — as various elections in various Democracies have been over the years. The average person is most likely to care about some but not all issues. And therefore some elections will be more important than others to that individual. Modi’s election was to people in North India in 2014, for instance. As was Jayalalitha’s in 2011. Given we can estimate the worth of each vote on an ongoing basis, wouldn’t the ideal option be to grant people a fixed number of votes for their life-time which is significantly lower than the number of elections they are likely to encounter over their life? And, better still allow them to trade it?

The benefit of this system is that the poor have an added advantage. One’d think. Unless one is wrong about it.

[1] – Did you notice that Clay Aiken contested and lost the Congressional race for the 2nd District in North Carolina?

[2] – Impact calculations were made for North Carolina along the same lines as we did here.

Studying the Test & Living the Predicted Life

There is a problem in our culture at this particular moment as it relates to mankind’s technological evolution. Data Mining has permeated most avenues of human activity; particularly those of readers of this post. The most common instances of it that we’d encounter are when we buy things online or watch something on Netflix. It’s also true of what we are likely to read — be it on Zite/Pulse or on the venerated New York Times — and what we see on the aisles of organised retail.

Of course it makes much sense and profit for those companies that’ve mined us; at least in the short term. The basic algorithm that is now about 20 years old is mind numbingly simple at its simplistic core: it merely discovers a rule for association with a certain level of confidence. That is, given you have bought X and Y you are likely to buy Z. Extend this for movies watched or articles read. The algorithms in those cases can be tweaked to something more fancy and tailored to one’s profile, but the recommendation problem is still Chapter 9 of this now classic textbook.

The problem as you can see is that the recommendation problem relies on a quasi closed set. That is, every subset of a frequent set has to necessarily be frequent itself. In other words, Z had to have been bought or watched or read enough times for it to be recommended to you. In the larger scheme of things, it simply means as a culture we’re now settling for exploit in the explore vs exploit problem that life presents us. Directors and producers who will invariably let this affect their art reduce the possibility of art transcending status-quo. It possibly also explains click baits on headlines and our reaction to it.

The other problem this creates is that it incentivises ghettoisation. For instance, one could argue, some 20 years ago the average person had a much higher likelihood of consuming news/commentary and general information from a wider variety of sources. Ever since this slide towards creating targeted content for specific groups started, with FOX News or Sun TV being good examples, the distinct constituencies have moved farther away from the original centre of public opinion. After all, once a bubble has been created it only makes sense for that bubble to feed itself. Its mean appears to be the true mean from within. And purists within will want to move the bubble ever farther away from their own and therefore the centre of public opinion. This process, as any of us who has a Twitter account can vouch for, has been exponentially accelerated in that part of media whose content is user generated.

The efficiency of tending to settle for exploit as opposed to explore makes sense for those selling. And makes absolutely none whatsoever for those buying. What we get is bad art, bad journalism, poor imagination in the design of products that’re sold and a far more intolerant society. A statistical understanding of the world is useful only when the world itself does not react to the statistical description of it. Since we’ve crossed that Rubicon, it’s reasonable to ask if Data Science as it applies to recommendation systems serves any positive purpose.