The Abundance of Big Data

The other day I was facing a long drive with nothing to read, so I went onto Audiobooks and grabbed the first thing that came up, Big Data: A revolution that will transform how we live, work, and think (2013) by Viktor Mayer-Schonberger and Kenneth Cukier, Houghton Mifflin Harcourt, Boston and New York.
http://books.google.ae/books?id=uy4lh-WEhhIC&pg=PR3&source=gbs_selected_pages&cad=3#v=onepage&q&f=false

I learned in the book about the algorithm by which Amazon is able to "recommend books to users based on their individual shopping preferences ...Amazon analyst Greg Linden saw a new way of doing things ... What if the site could make associations between products themselves rather than compare the preferences of people with other people? In 1998 Linden and his colleagues applied for a patent on ‘item-to-item’ collaborative filtering and the shift in approach made a big difference – a big data difference."
Quoted from the book in this blog post:
http://brandlogik.com/of-mice-and-men-your-big-data-big-brand-moment/

Linden is himself quoted in the book as saying the ideal algorithm would not show you dozens of recommended books but only the very book you were next going to buy. This is exactly what happened when I went onto the site, saw the first book offered, recalled having heard about it on http://democracynow.org, possibly in connection with Edward Snowden's revelations of NSA spying, and in a click dealt Audiobooks another data point.  The fact that Audiobooks would be able to pinpoint my interests so accurately suggests that NSA is not the only entity interested in my data. The fact that we collectively take this and all Facebook knows about us in stride (and Amazon, Google, Wallmart, and any given phone provider etc.) shows how much we accept this as normal behavior, and the book Big Data details how pervasive and normal this is. In fact, most of us accept websites tracking us as a fair trade, our data for their free services. We are only slightly annoyed when we find that corporations are doing this extensively, as when Apple was found to be tracking user movements via the GPS on their newly purchased iPads without their knowledge (as reported in the book).

Transparency is in fact the issue here.The problem with NSA spying, as Michael Geist points out, is that the government conceals and dissimulates about what they are doing with their harvest of big data. Writing in the Canadian context, he reports where a Canadian 'official' "remarked that in the wake of the Snowden revelations the political risk did not lie with surveillance itself, since most Canadians basically trusted their government and intelligence agencies to avoid misuse. Rather, the real concern was with being caught lying about the surveillance activities. This person was of the view that Canadians would accept surveillance, but they would not accept lying about surveillance programs."
http://www.michaelgeist.ca/content/view/7062/125/

Canada's neighbor to the south has not instilled confidence in its government's integrity lately, but that aside, the book Big Data is mind opening in explaining how that government's approach to data mining is not at all unusual, is in fact the norm for use of the abundance of data available in our era, and is certainly what we can expect more of in the future.

The book explains the shift in statistical analysis that big data has evoked. In the past, when data were tediously collected and analyzed, the empirical approach was to form a hypothesis and attempt to then support that hypothesis by constructing an experiment to establish causality from one variable to the next through random sampling, and extrapolate that out over larger populations.  Random sampling was shown to be reasonably reliable, where N size was large enough, to make predictions accurate for the population at large.

However, where the availability of data approaches infinity, and N equals "all" (all available data can be aggregated and analyzed through computer algorithms) then it turns out the approach to research is not to form a hypothesis at all, but to examine correlations in the data and see what patterns emerge. Thus the emergent approach to research in education, to take the instance that is the topic of this blog, is not toward replicating and inventing new experiments with inevitable shortcomings in data collection methods, where extrapolability to wider populations is always in doubt, but toward harvesting as much data as possible and seeing what pops out, as practiced with "learning analytics".

Where the number of data points is massive, and the amount of data is almost limitless, the results produced this way are exceedingly predictive, to the point where real-time pictures of happening phenomena (like the spread of flu outbreaks) can be inferred through correlating data points, and to where it is getting impossible to compete in markets without having the edge over rivals on data aggregation, storage, and algorithms for analysis.

Big Data takes pains to point out that correlation does not imply causality (it is what it is; when this and that are present then something else tends to happen as well, and the data show where this has historically been true, though they do not tell us why or how). However, it is possible to arrive at hypotheses to explain observed trends and then continue to observe that subsequent data support that hypothesis. For example, Ray Kurzweil has collected copious data to support the contention that technology improves on an exponential curve which on closer examination is seen to be comprised of repeated S movements as paradigm boundaries are crossed. This prediction is akin to Moore's Law which stated (in 1965) that the number of transistors on integrated circuits doubles approximately every two years, and this has proven to be the case ever since. Kurzweil postulates that from such data computers should move beyond human comprehension at a point called Singularity, which is predicted as early as 2030, or by Kutzweil's reckoning, 2045 (more information on Wikipedia and at http://www.singularity.com/, and in Kurzweil's words in a TED Talk, below).



However, lines can be crossed. The point is made more than once early in the book Big Data, and elaborated on in a later chapter, that such analysis can help authorities predict who will commit crimes before they happen. If arrests (or assassinations) are carried out on the basis of such models, is this itself a crime, a violation of supreme law of the land? On reading this book, it seems more in context now why governments venture toward this grey area in an era where all sides are seeking to leverage big data, or risk being one-upped (though some matters of conscience and justice remain unchanged, or should, and therein lies the conundrum). In its last chapters, Big Data explores the risks and implications for individual freedom and privacy.

Learning2gether with Everybody

Do you ever feel you are verging on giving more time to your online endeavors than you feel you can use productively in your face-to-face ones? Whether or not that's actually the case (I haven't undertaken a systematic time analysis) I sometimes feel that way. That problem prompted one of my connections, frustrated in doing it himself, to run out of time and send me a notice to be posted on his behalf to one of my networks, and I responded with help for him to troubleshoot the problem. The time it took for that was probably more than if I had simply posted his notice, but posting on behalf of others as community leader could imply endorsement, and it's best if everyone in the community is enabled to work independently. But mainly, as I explained in my reply, I encourage independence because we all only get so many keystrokes in a day.

The above has been an aside by way of introduction, but I have been thinking to document one aspect of online community steerage that consumes a lot of those keystrokes; i.e. making announcement on social media sites. Intelligent use of tagging, and exploiting scripts and connections between social media sites might help to attenuate the problem, but writing it out might help me to see where there is potential for that, or potentially of even more value, maybe someone will comment with a useful solution to the problem.

One problem is that the social media landscape changes so often. I became aware a couple of years ago that social media specialists and consultants were being hired by entities seeking to manage their social presence (not only in getting out the message but also the quality of their footprint) but it is only recently that the abundance of social spaces that people inhabit has got to the point, for me at any rate, where it is running up against that finite number of key presses you get in a given day. Those consultants must earn their pay, if only in compensation for carpal tunnel.

Let's take for example the next Learning2gether event which is coming up in a few days, and I need to get the word out. Learning2gether is organized through a wiki, which means that a community can contribute key strokes to entering the events, but in practice those keystrokes are mostly mine. So most of what you see at http://learning2gether.pbworks.com is my own input, though occasionally that of others (and much appreciated!).

So the events themselves are shaped at that wiki, and when it's time to announce them, I scoop out the text and copy it into a Notepad on my PC, from where I can fashion versions to be sent out to various social networking sites. If we are planning to use HoA (Hangout on Air) I then set up http://webheadsinaction.org/live with an announcement of the upcoming event. I use that page to keep our connection with the http://worldbridges.net/ and http://edtechtalk.com/ communities current.

One place I post it is here: http://fourc.ca/calendar/. Tyson Seburn has worked with Learning2gether in the past - on Monday, May 6, 2013, we helped him host TESL Toronto presents: Aga Palalas – mobile apps for language learning, http://learning2gether.net/2013/05/06/tesl-toronto-presents-aga-palalas-mobile-apps-for-language-learning/. His calendar is not the ultimate solution to the world's educators' pooling in one place a comprehensive listing of all online and f2f professional events of interest to them (such a feature would be a script that goes into the wild and harvests all such notices tagged with the tag it is looking for; spam could be prevented by people posting such notices registering with the script, as with Stephen Downes's gRSShopper: http://grsshopper.downes.ca/index.html) but http://fourc.ca/calendar/ is at least easy to use manually, and events posted end up on the calendar.  I learned about it from Graham Stanley's posting here:
http://blog-efl.blogspot.ae/2014/01/tracking-conferences-and-connecting.html

Then I'll post to relevant Nings. I don't use Ning much any more (here's why: http://evomlit.wordpress.com/2011/07/11/thanks-pearson-and-ning-but-it-just-doesnt-work/) but if the event is related to a Ning that is worthwhile and is supported by an institution that will pay for it, I post to that one. For example I post all L2g events at the TESOL Arabia EdTech SIG page at http://taedtech.ning.com/.

Next I'll post an event on the relevant Google+ Community. This pushes it out to all subscribers at that community, and it can be shared (as an event) with one other community. I don't understand why just one, though it's possible to initiate the event elsewhere and share that with another community, thus getting your event out to 4, or to 6 or 8, but this cuts into our daily ration of keystrokes. I can understand the implications for flooding communities with events, but as a responsible user, I would prefer to make that decision (and let Google decide for all the irresponsible users :-). Ok, we've enjoyed my painting myself into a corner, and since I don't like to overdo the events, I simply copy and paste the descriptions of the event into "Share what's new" in a number of other Google+ Communities.

Here are some of my own communities:

• MultiMOOC - http://gplus.to/multimooc
• Learning2gether - http://gplus.to/learning2gether
• EdTech Mojo - http://gplus.to/edtechmojo
• Webheadsinaction - http://gplus.to/webheadsinaction

Next comes Facebook.  I post announcements on the relevant FB groups, and here, as with G+C's I'm careful to include the relevant #tags. The posting itself is relatively easy; normally I just copy and paste what I put into my G+C's to each group.

Again, some of my communities

• Multiliteracies - https://www.facebook.com/groups/evomlit/
• Learning2gether - https://www.facebook.com/groups/learning2gether/
• Webheads - https://www.facebook.com/groups/webheadsinaction/
• TESOL Arabia EdTech SIG - https://www.facebook.com/groups/TAEdTech/

Indeed, reading this, I can see that I need a script that posts from one place into FB (not to my main page, but to the groups I specify) and same for G+ Communities I specify. Many social networks allow you to post on Twitter and to your wall at FB at the time you make a post on that network. I notice that a lot of my colleagues do this. Perhaps someone will remind me of the killer app that will do just what I want it to, directed at just the communities and groups I think will appreciate the information (or offer to help me code one).

Finally, I post to the Yahoo! Groups that have held their communities together for a long time and whose community members often support Learning2gether.  The two that I maintain are:

• Webheads in Action - http://groups.yahoo.com/neo/groups/evonline2002_webheads/ 
• Multilit - http://groups.yahoo.com/neo/groups/multilit/

Personally I feel that Twitter is most effective nearer the time of the event so I don't usually post to Twitter until the event is nigh, though after the event I'll move its archive to http://learning2gether.net/ and erase it from http://learning2gether.pbworks.com/w/page/32206114/volunteersneeded. Once it's archived I'll Scoop.it here: http://www.scoop.it/t/learning2gether, and let that one send a tweet, or to FB if it was my presentation.