Can AlphaGo Help You Stay Alpha Dog?


The recent triumph of AI program AlphaGo playing against a human, signals just how far advanced analytics has come. What lessons can you learn to get a competitive edge for your business?

LittleBlogAuthorGraphic  David Hodgson, March 15, 2016

Almost two decades ago, in 1997, IBM’s Deep Blue chess playing computer beat the reigning world champion, Garry Kasparov, in a six game match under tournament conditions. The world realized, perhaps for the first time, that HAL of “2001 A Space Odyssey” fame, was going to arrive at some point, though a few years later than cast by Kubrick.

Then, in 2011, IBM’s Watson computer stunned us by winning at Jeopardy. If you haven’t actually seen Watson playing Jeopardy click on that YouTube link; it’s truly awesome. The feeling of invasion is greater seeing Watson, perhaps because we can all imagine playing Jeopardy, and the question and answer approach is so “human”.

Which brings us to current events. Google’s DeepMind research team has developed AlphaGo which beat Fan Hui 5-0 last October. Hui is the current European Go champion and 2 dan master. This was impressive enough, but today saw AlphaGo win 4-1 playing Lee Sodol, the current World Champion, a South Korean 9 dan Grandmaster.   Send Lee Sodol a message of support somehow, because being on the coalface of human defeat by computers must be tough.

What is happening here?

Closed system games like Chess and Go are complicated, but have simple rules and a known, although massive, number of variables. There are more possible Go board move sequences than the estimated 1080 atoms in the visible universe. This is a formidable problem, but a different sort to the open ended question and answer format of a game like Jeopardy

AlphaGo’s algorithms use a combination of value-weighted, Monte-Carlo tree search techniques and a neural network implementation. The DeepMind team’s approach to machine learning involved extensive training from both human and computer play. AlphaGo played itself to rapidly learn the outcomes of numerous different options.

Watson used Hadoop to store masses of unstructured information, including the entire text of Wikipedia, that it could search with analytical techniques in real time. Equally significant in Watson’s case is that it was responding to natural language questions that it had first to understand using similar search techniques.

A third powerhouse for change, Facebook is also experimenting with AI systems and has their own Go-playing system Darkforest, also based on combining machine learning and tree search techniques.

Between them and the numerous other AI projects underway in different domains, we have the building blocks for HAL’s arrival.

So what?

I hear some saying “So what David?”. “This is interesting to learn about, and with the election I had missed it in the news, but of what importance is it to me?”

DeepMind is targeting smartphone assistants, healthcare, and robotics as the practical outcome for their experimental work with AlphaGo. From their website:

“The algorithms we build are capable of learning for themselves directly from raw experience or data, and are general in that they can perform well across a wide variety of tasks straight out of the box.”

IBM has already applied versions of Watson to practical problems, offers it as a service for anyone to buy and a developer community to encourage experimentation. An example of a practical application is the partnership with Sloan Kettering to fine tune cancer treatment. Similarly DeepMind is partnering with the UK’s National Health Service to improve its services.

Although for specific solutions much secret sauce is often preserved, the framework of these systems is usually Open-Source software. An important component of Watson is the Apache Unstructured Information Management Architecture (UIMA) software. These same tools and techniques will be what disrupt your business soon and you will want to be an early adopter.

Fed with the right data, a Watson-type system could answer new questions that nobody yet knows the answers too. Or applied to real-world problems an AlphaGo-type system could decided on the best course of action given many variables and alternatives. Leading the field in practical solutions IBM calls this ‘cognitive business’ and it is definitely a part of our future.

You Control your Future

In the panorama of the Digital Transformation, AI is out there as a wildcard with seeming limitless possibilities. We are both familiar with, and scared of, these futures because of numerous science fiction dramas. HAL is not here yet, but its coming. For you it’s really a case of whether your company or the competition deploy machine learning systems first. You don’t need an AI system to answer that question.


Image credit: NYTimes




Data Makes the World Go Round

Big Data

The key to success in the digital age is transforming your business model by leveraging data in a way that either improves your operations or adds value to your customers. Unlike money, much of the data that makes the world go round is free! Where can you get it and how could you use it?

 LittleBlogAuthorGraphic  David Hodgson, March 7, 2016

A year ago I posted on the idea that “data is everywhere”. Since then there is more data in the world, greater accessibility and better tools. Read that old post if you have time because it will still add something to what you learn here if you are a newcomer to Big Data Analytics.

What Data?

Of course your company has been using data for years and doing analysis and reporting on it too. With an explosion of new analytical tools and the low cost of cloud based facilities and storage the world has discovered that the business data you have had in structured databases for years is just the tip of the iceberg. With so many recreational and business activities now being conducted on-line, the world is generating lots more data and this data, often very unstructured and fragmented, is becoming both accessible and useful.

This data might in fact be in-house, in system logs and Excel spreadsheets i.e. not readily available for large scale analysis. But it might be outside the company and a new asset that you could acquire and derive value from. It could be as crazy sounding as data from social media feeds (like Twitter or Facebook), or it could be more down to earth in terms of semi-structured lists. Most interesting is that some of it could be free!

To find some of this “new” data you might start by just googling “free data sources”. This will yield many references to follow up, but some of the richest data and most useful to business is provided by government agencies.  The list below will lead you to a wide variety of examples.

Sources of Free Data from Government Agencies

US Government’s Open Data
US Census Bureau 
National Climatic Data Center 
The CIA World Factbook 
Socrata – Open Data Network
European Union Open Data Portal 
UK Open Government Initiative
NHS Health and Social Care Information Centre 
Government of Canada Open Data
Open Government Data Catalogs 
UNICEF Statistics and Monitoring
World Health Organization

But how do I use it?

Using data from these or other sources is a multi stage process. First you have to access the data, be able to bring it into your repository, maybe Hadoop. Then you must clean it up or format it so that it fits for your use (key value pairs, JSON, structured, semi-structured etc.). Lastly you have to create the analysis and and visualization that will give you the valuable insights you seek, often by pairing it with your existing structured data.

Accessing data could be a simple Extra, Transform & Load (ETL) process if you have access to the source database. Often though the data you want is only accessible through a defined Application Program Interface (API ).  The Socrata link above details APIs for all of the thousands of data sources listed (e.g CA Prop 39 grants)

The API is the underpinning of most apps in the digital economy and the way apps interact or are integrated. Having a programming team that can use publicly available APIs to access data is an essential step for you. One of the hubs of information on APIs is the Programmable Web site. If you are building your analytics app on a hosted platform like AWS, Azure or Google they have databases of useful data that you can access for free through their APIs. For example see the list of AWS public datasets

But how could any of this be useful to me?

Firstly you need to educate yourself about the potential for Big Data analytics and the availability of data from numerous sources. You don’t know what you don’t know until you start to look at what is going on. I have heard stories of banks detecting ATM weaknesses in their competitors through social media feeds and using that knowledge to target new customers. Or the insurance company that gets a red flag on a fraudulent claim by spotting that the two opposing parties in a claim have been Facebook friends for years.

Perhaps these examples of how some government agencies are using data will start the thought process. Or read this report from McKinsey for results from the world of business.

Once you have the general picture then it’s the questions you want to answer that are more important than finding data. Step back and think about questions that, if you could answer them, would give you some sort of operational advantage or competitive edge. Make sure that you know what you would do with the answers if you had them!! Then hire a good data scientist or two and start a first project to amaze yourself. They will find the data to answer the questions and work out how to do the analysis and visualization.

What can go wrong here?

Many things could go wrong, but I will highlight three:

1. You spend too much and go over budget

There’s no such thing as a free lunch right? and processing even free data is going to be time consuming and costly as you hire the right people, buy the right tools and experiment to get a useful result.

2. You under use the data and fail to get full value

The real value of your analytics will be realized as you allow business users to interact with the data in the context of making day-to-day decisions. Limiting access just to the data scientists who started the initiative will be a big mistake.

3. You forget the need for security & compliance

Probably the biggest thing that can go wrong, especially as you expand access to business users, is falling foul of privacy and compliance regulations. For Hadoop based systems you will need to leverage security systems like Ranger, Sentry and Knox to pass user information up to the application layer, and enforce access authorization based on data lineage.

Get Going

Data really does make our digital world go round. If you haven’t yet found new data sources and new ways to use data to super charge your business, then you may be behind your competitors. Get going, find a project to start a “Big Data” initiative and start your own Digital Transformation.


Image credit: Unknown origin