The Limits of Data December 21, 2020
Posted by Peter Varhol in Algorithms, Machine Learning, Strategy.Tags: bias, data
1 comment so far
I’ve been teaching statistics and operations research since, well, the mid-1980s I guess, to more or less degrees of student sophistication. In most cases, I try to add some real world context over what most students consider to be a dry and irrelevant topic, even as I realize that most people are in the room because it’s required for their degree.
Except that over the last few years statistics and analytics has shown itself to be anything but irrelevant. As data has become easier to collect and store, and faster processing has brought information to life from data in real time, more and more scientific, engineering, business, and management professionals are at least trying to use data to make more justifiable decisions.
(I casually follow American professional football, and have been amazed over the last few years to see disdain for any sort of analytics turn into a slavish following and detailed definition of obscure analytical results.)
And at least some people seem to be paying attention. I still get a lot of “I’m not a math person” or “I make my decisions without considering data” but that is becoming less common as people recognize that they are expected to justify the directions they take.
In general this is a good trend. An informed decision is demonstrably better than one based on “gut feel.” As the saying goes, you are entitled to your own opinion, but not your own facts. Professionals making decisions based on analytics won’t always result in the right answer, but it will be better than what many are doing today.
But data is not a universal panacea. First, any data set we use may not accurately represent the problem domain. There may have been data collection errors, or the data may not be highly related with the conclusion you want to draw. For example, there may be a correlation with intelligence and income, but the true determiner may well be education, not intelligence. In these circumstances, our analytics can lead us to the wrong conclusion.
Our data can also be biased. Machine learning systems do a poor job at facial recognition of other races, for example, causing high levels of misidentification. This is primarily because we don’t have good data on facial characteristics of those races. Years ago, Amazon came up with an algorithm to identify potential candidates for IT jobs that overwhelmingly used male data. The algorithms quite naturally came to the incorrect conclusion that only men made good IT workers.
So while our data can make decisions more accurately, it’s only the case when we apply it correctly. And that’s not as easy as it sounds.
Cognitive Bias in Machine Learning June 8, 2018
Posted by Peter Varhol in Algorithms, Machine Learning.Tags: Algorithms, bias, Machine Learning, neural networks
add a comment
I’ve danced around this topic over the last eight months or so, and now think I’ve learned enough to say something definitive.
So here is the problem. Neural networks are sets of layered algorithms. It might have three layers, or it might have over a hundred. These algorithms, which can be as simple as polynomials, or as complex as partial derivatives, process incoming data and pass it up to the next level for further processing.
Where do these layers of algorithms come from? Well, that’s a much longer story. For the time being, let’s just say they are the secret sauce of the data scientists.
The entire goal is to produce an output that accurately models the real-life outcome. So we run our independent variables through the layers of algorithms and compare the output to the reality.
There is a problem with this. Given a complex enough neural network, it is entirely possible that any data set can be trained to provide an acceptable output, even if it’s not related to the problem domain.
And that’s the problem. If any random data set will work for training, then choosing a truly representative data set can be a real challenge. Of course, will would never use a random data set for training; we would use something that was related to the problem domain. And here is where the potential for bias creeps in.
Bias is disproportionate weight in favor of or against one thing, person, or group compared with another. It’s when we make one choice over another for emotional rather than logical reasons. Of course, computers can’t show emotion, but they can reflect the biases of their data, and the biases of their designers. So we have data scientists either working with data sets that don’t completely represent the problem domain, or making incorrect assumptions between relationships between data and results.
In fact, depending on the data, the bias can be drastic. MIT researchers have recently demonstrated Norman, the psychopathic AI. Norman was trained with written captions describing graphic images about death from the darkest corners of Reddit. Norman sees only violent imagery in Rorschach inkblot cards. And of course there was Tay, the artificial intelligence chatter bot that was originally released by Microsoft Corporation on Twitter. After less than a day, Twitter users discovered that Tay could be trained with tweets, and trained it to be obnoxious and racist.
So the data we use to train our neural networks can make a big difference in the results. We might pick out terrorists based on their appearance or religious affiliation, rather than any behavior or criminal record. Or we might deny loans to people based on where they live, rather than their ability to pay.
On the one hand, biases may make machine learning systems seem more, well, human. On the other, we want outcomes from our machine learning systems that accurately reflect the problem domain, and not biased. We don’t want our human biases to become inherited by our computers.
Bias and Truth and AI, Oh My October 4, 2017
Posted by Peter Varhol in Machine Learning, Software development, Technology and Culture.Tags: AI, bias
add a comment
I was just accepted to speak at the Toronto Machine Learning Summit next month, a circumstance that I never thought might happen. I am not an academic researcher, after all, and while I have jumped back into machine learning after a hiatus of two decades, many more are fundamentally better at it than me.
The topic is Cognitive Bias in AI: What Can Go Wrong? It’s rather a follow-on from the presentations I’ve done on bias in software development and testing, but it doesn’t really fit into my usual conferences, so I attempted to cast my net into new waters. For some reason, the Toronto folks said yes.
But it mostly means that I have to actually write the presentation. And here is the rub: We tend to believe that intelligent systems are always correct, and in those rare circumstances where they are not, it is simply the result of a software bug.
No. A bug is a one-off error that can be corrected in code. A bias is a systematic adjustment toward a predetermined conclusion that cannot be fixed with a code change. At the very least the training data and machine learning architecture have to be re-thought.
And we have examples such as these:
If you’re not a white male, artificial intelligence’s use in healthcare could be dangerous.
When artificial intelligence judges a beauty contest, white people win.
But the fundamental question, as we pursue solutions across a wide range of applications, is: Do we want human decisions, or do we want correct ones? That’s not to say that all human decisions are incorrect, but only to point out that much of what we decide is colored by our bias.
I’m curious about what AI applications decide about this one. Do we want to eliminate the bias, or do we want to reflect the values of the data we choose to use? I hope the former, but the latter may win out, for a variety of reasons.
The Future is Now June 23, 2017
Posted by Peter Varhol in Algorithms, Technology and Culture.Tags: Algorithms, bias
add a comment
And it is messy. This article notes that it has been 15 years since the release of Minority Report, and today we are using predictive analytics to determine who might commit a crime, and where.
Perhaps it is the sign of the times. Despite being safer than ever, we are also more afraid than ever. We may not let our electronics onto commercial planes (though they are presumably okay in cargo). We want to flag and restrict contact with people deemed high-risk. We want to stay home. We want the police to have more powers.
In a way it’s understandable. This is a bias described aptly by Daniel Kahneman. We can extrapolate from the general to the particular, but not from the particular to the general. And there is also the primacy bias. When we see a mass attack, was are likely to instinctively interpret that as an increase in attacks in general, rather than looking at the trends over time.
I’m reminded of the Buffalo Springfield song: “Paranoia strikes deep, into your lives it will creep.”
But there is a problem using predictive analytics in this fashion, as Tom Cruise discovered. And this gets back to Nicholas Carr’s point – we can’t effectively automate what we can’t do ourselves. If a human cannot draw the same or more accurate conclusions, we have no right to rely blindly on analytics.
I suspect that we are going to see increased misuses of analytics in the future, and that is regrettable. We have to have data scientists, economists, and computer professionals step up and say that a particular application is inappropriate.
I will do so when I can. I hope others will, too.
Being a Curmudgeon Has its Benefits August 1, 2016
Posted by Peter Varhol in Technology and Culture, Uncategorized.Tags: bias
2 comments
I occasionally wax personal in my blog, as I did a year ago when I was facing a serious cancer diagnosis (the diagnosis was ultimately incorrect, and I am healthier than ever). Occasionally I just have to say something about a particular moment, whether or not it relates to my target blog topics.
This morning I got a regular email newsletter from Marc Cendella of The Ladders, a job search service for salaries over $100K. The title was “When the kid interviewing you says you’re too old…” In it, Cendella says that age discrimination in hiring is prevalent, and offers the older job seeker a checklist of items to attempt to overcome that bias.
Here is where I call a foul. Certainly there are things that a job seeker can do in order to make him- or her self appear to be a better fit for a given job. In general, those things range from the common-sensical (be engaged and current in your profession and energetic in your life pursuits) to the absurd (facelifts and hair coloring).
But it’s a two-way street. Why not also suggest to the hiring managers that they might have a bias that is not well serving their organization, and how they might recognize and correct that deficiency?
Oh, that’s right. Businesses like The Ladders make money from those companies doing the hiring, not from job seekers. The Ladders would rather tell the job seeker to change, rather than the hiring manager.
I would imagine that in a lengthy career spanning a dozen or more jobs and dozens of interviews, I have experienced some types of bias and discrimination. Probably everyone has; we tend to form initial impressions of someone we just met in under a second, and those first impressions can be both unconscious and difficult to overcome.
Bias in hiring is particularly difficult to demonstrate, as there could be any reason or no reason to not be selected for a job. The prospective employer certainly isn’t telling (usually), so most of this left to speculation or inference, and not even worth considering, let alone actionable.
But I found this newsletter from The Ladders to be singularly offensive. I instinctively interpreted it as “It’s not my problem that I am biased, it’s yours in that you are too old.” I deeply resent that Cendella says that it’s a problem for job-seekers, rather than a problem for hiring managers (or for both). If hiring managers let such biases creep into their decision process, they are doing both themselves and their organization a serious disservice.
I have always been sanguine about bias in hiring. My attitude has been that if I am discounted because of a personal characteristic outside of my control, it’s a place I probably wouldn’t want to work at anyway.
The fact of the matter is that unless we die young, or hit the jackpot, we are all destined to become older workers. Everyone, deal with it.
Cultural Fit is Bullshit January 23, 2016
Posted by Peter Varhol in Technology and Culture.Tags: bias, education
1 comment so far
This post has spent months (maybe years) in the making. And yes, I like the rhyme of the title. I revisited it yet again after reading this article on minority hiring in Silicon Valley in Bloomberg News.
I am not a mainstream techie. I earned two degrees in psychology before turning to programming and technology in general, in my late 20s. I am not a very good techie. I have a solid foundation to understand and explain concepts (I was a university professor at one point), but never more than an average coder. Nevertheless, I’ve made mostly a decent living in technology, though not in Silicon Valley. Just so you know where I’m coming from.
I used to believe that cultural fit was the preeminent job requirement. Now I understand that’s what the employer would like me to believe. They could hire or fire on a whim, rather than what they actually need. In fact, whether or not I could do that job has no impact on my hireability.
So minorities (and almost certainly others who don’t fit into pre-established norms) are at a disadvantage because they didn’t start coding when they were seven? This is where the bullshit starts. Does that make them better coders? Possibly, although certainly not provably. Does that make them better contributors? Now there is the rub. I would argue no.
But we are befuddled by candidates who are savants at placing bits of data into processor registers and making it do backflips. That is a worthwhile skill, but it’s not the only skill necessary to succeed, as an individual, as a part of a team, and as a company. Even in Silicon Valley. If your teams are all A-list coders, you are missing out on some essential skills. Yet you seem to be fine with that.
In my health issues over the last year, I was fortunate to encounter a couple of doctors who treated me as a person, rather than as a collection of symptoms. I challenge Silicon Valley to do the same. Understand at a deep level what your teams need, and interview and hire based on those needs. Understand not only the technical skills, but the social dynamics and complementary skills that are necessary for any team to succeed. You are not doing so.
The mantra of cultural fit has enabled Silicon Valley to ignore deeper issues of team dynamics, skills needs, and what drives people to be successful. You hire people like you. Or people that fit into a predetermined slot. I get it, but you refuse to get out of your comfort zone to look at what might make you successful. You are blind. And, in the kingdom of the blind, the one-eyed man is king, if I may quote Desiderius Erasmus.
I have no right to do so, but I challenge Silicon Valley. Yes, you.
Cognitive Bias and Regression to the Mean April 29, 2014
Posted by Peter Varhol in Software development.Tags: bias, regression to the mean, testing
add a comment
We prefer to assign causality to events in our own lives, and in the world in general. If something positive happens, we tend to credit our intelligence, or dedication, or some other quality. If negative, we often blame others, or perhaps blame our own failings. Every day when the stock market closes, we read about how stocks have gone up or down for some perfectly understandable reason.
Bull. Most things in our lives and in the world don’t happen for a reason, or at least any reason we can readily identify. Our good fortune may be only peripherally related to our intelligence or abilities, and our bad fortune may simply arise from being in the wrong place at the wrong time.
Regression to the mean is simply one example of our need for causality, and how it results in bias. If we perform exceptionally well, we come to believe our own press releases, and behave as though we are high achievers. We might well be, but achievement is a slippery thing; it might disappear in a heartbeat.
Regression to the mean is a statistical concept. It simply notes that any time you get an exceptional result, it is unusual. Subsequent results are more likely to be closer to the average. It’s a concept often found in the natural sciences. For example, I am taller than either of my parents, so it is likely that my children (if I had any) would be shorter than me, since I am taller than many of my direct ancestors.
Applied to our lives, regression to the mean refers to the fact that what we do is a combination of skill and luck. We have little idea how much is skill, and how much luck. When we do exceptionally well at a task, we tend to attribute that to skill. When we do poorly, we often blame bad luck. Instead, exceptional performances are random (and rare) chance.
You can certainly argue that such a statistical concept doesn’t really apply to individual efforts, but I think the general principle holds. Sometimes we simply do better than other times, and it’s not clear that it reflects skill any more than (good or bad) luck.
Applied to software development and testing, regression to the mean gives us preconceived notions of the performance of the software based on who works on it. It makes us believe certain things about software based on the perceived superiority or inferiority of the team members based on our experiences.
The Role of Heuristics in Bias April 24, 2014
Posted by Peter Varhol in Software development.Tags: bias, heuristic, testing
add a comment
A heuristic is what we often refer to as a “rule of thumb”. We’ve experienced a particular situation on several occasions, and have come up with a step-by-step process for dealing with it. It’s purely System 1 thinking in action, as we assess the situation and blindly follow rules that have worked for us in the past.
And heuristics are great. They help us make decisions fast in situations that we’ve experienced in the past. But when the situation only appears similar, but is really different, applying our heuristic can have a very bad effect, if it’s not right.
Here’s a real life example. Years ago, I took flying lessons and obtained my pilot’s license. One of the lessons involved going “under the hood”. The hood is a plastic device that goes over your head (see image). When the hood is down, you can’t see anything. When the hood is raised, you can see the instrument panel, but not outside of the plane.
While the hood was down, the instructor pilot in the right seat put the plane into an unusual situation. That might be a bank, or a stall, or something that was unsustainable. When he raised the hood, I was required to use the instrument panel to analyze and diagnose the situation, and recover from it.
After several of these situations, I had developed a heuristic. I looked first at the turn and bank indicator; if we were turning or banking, I would get us back on course in straight flight. Then I would look at the airspeed indicator. If we were going too slow, I could lower the nose or advance power to get us back to a cruise speed.
This heuristic worked great, and four or five times I was able to recover the aircraft exceptionally quickly. I was quite proud of myself.
But my instructor figured out what I was doing, and the next time I applied my heuristic, it seemed to work. But I was fighting the controls! It wasn’t straight and level flight. I started scanning other instruments, and discovered that we were losing over a thousand feet a minute.
At that point, my heuristic had failed. But I wasn’t able to go back and analyze the situation. My mind froze, and if it weren’t for the instructor pilot, we may well have crashed.
The lesson is that when your heuristic doesn’t work, it may be worse than starting over at the beginning. You may simply not be able to.
Cutting Edge Computing 