This blog is a simple introduction on Item Response Theory. So what we’re going to look over here is what is this S curve? What is item response theory? Just to understand at a very high level, what are the applications and how IRT is involved in it? Where do we come across all of these things? And why should you as a product manager, as an instructional designer, as a learning outcomes engineer, even care for this particular thing? And also why NASSCOM is called a qualification pack?
- WORDS:3626
- TYPE:Post
- VIDEO:1
- SLIDES:None
- TOPIC:Humans Of EdTech
- TIME:19 Minutes
Hi guys so welcome to another episode of Humans of EdTech. So today we are going to look at something called as the Item Response Theory. I mean it may sound a little intimidating but it is something that you are using or you’re used it in the past but you don’t know you’ve used it or maybe you are designing it for your learner’s if you’re a product manager, if you’re an instructional designer, you’re probably have plans to use this for your learner’s.
About S Curve
So let’s have a look at what is this item response theory, why the S curve matters. So typically, I think we always are fascinated by the bell curve of normal distribution from either side of the bell curve and the bulk of the majority somewhere in the centre.
So here I think we move from the bell curve to an S curve and I think why it really matters. So the objective of this particular video is not to go into the mathematical depth of item response theory or it’s, I think one of the simplest, gentlest introductions.
And we just want to not throw any psychological jargon or anything of that sort, but just to very gently introduce this thing to you.
So let’s quickly kind of, you know, get started with this particular part. Right? So what we’re going to look over here is what is this S curve? You know, what is item response theory? Just to understand at a very high level, what are the applications? Where do we come across all of these things?
So, let’s quickly jump into it. So I’m sure you must have seen this particular curve many times in the past. So this is something which has wide applications whether it is in performance appraisal, and a lot of phenomena in and around the nature like height of students, distribution of various things is typically the bell shaped curve, where a vast majority is in the centre and the the outliers or so to speak the people who are at the extreme end or extreme end of the performance in a value chain are on the either side.
So this is something I’m sure you must have seen and have a lot of examples of a lot of phenomena which follow this kind of distribution.
Today, we are going to look at something that we call as the S curve. So, in the S curve what we are going to look at is there can be you know, a family of these S curves where we are essentially looking at building out these s curves where on the left hand side you’re able to see three s curves which have similar slopes, but they are placed on different sites.
They have similar slopes, but they are placed a little differently and spaced out Whereas on the right hand side, you may be able to see three curves, where the median, so to speak, and let’s just call it medium. They all intersect, they all pass through a single point. But their slopes are different.
They all pass through the same point, but their slopes are different. So if you look at a lot of phenomena in the nature of like if you’re doing some sort of ads, and you’re doing some sort of a newspaper impressions on top of it, newspaper ads, so if you do like one ad to two AD, nothing is going to happen.
But I think if you’re going to do ads enough number of times, then I think the awareness is going to build up. But if you do ads an obscene number of times, then I think it is going to get saturated. So I think there is a natural limit. I think initially nothing picks up then things really pick up in a hurry, and then it kind of saturates.
So whether it is outdoor advertising, you know all those pole banners. That you have, or a lot of other phenomena users. So depending on where you are coming from and what your background is, you may be able to see step function over here if you are from engineering background, which is, you know, not exactly clean step function, or if you have a very statistical background, you might be able to see some sort of cumulative density function, or anything of that sort.
But what this essentially says is that, if you look at any of these curves, it essentially divides into, let’s just say the left hand and right. And then there are some curves, like say over here, the blue curve divides the space in left and right much more strongly, it has a slightly higher discrimination, so to speak, and I think we’ll come to some of these things a bit later.
But where is this green line? It’s a little bit more, it divides the same, you know, in left and right, but it’s like the It’s not a very hard division, I’m just trying to be very human with respect to your choice of vocabulary and communication.
Well, I just want to say that the blue line or the black line, as you can see over here, it divides vertically in left and right on the green line doesn’t divide vertically very strongly. So that’s essentially the thing. And we’ll come to this particular thing.
And the core and the height of the item response theory is understanding the mathematics behind this particular curve. And if you’re able to understand this particular part, if you’re just able to understand these two particular charts and diagrams, not maybe mathematically, but just conceptually, or getting the intuition behind it, you will have a very good conversation and understanding of what item response theory is.
So let’s quickly get some get started with what item response theory is. So this is something that I’ve picked up from Wikipedia. And I just want to, you know, you can read through the definition and everything. But what I really want to point your attention to here is just a couple of things.
If you’ve taken any of the adaptive tests if you’re taking a CAT for getting into some sort of MBA School in India GREG MAT or any other aptitude test, you most likely, or possibly have seen the item response theory at play. You’ve seen some of that anywhere, I think you’ll find some sort of standardisation of test, uniform scoring, whether you’re taking the test from this particular country or the US or any of the African countries or somewhere in the southeast or somewhere in the Far East.
The GRE essentially stays the same everywhere. A standardized test so you possibly saw some sort of item response theory being at play over there.
So I think I just put the definition because I think the reason being there is a lot of You know, I would not say facade, but I think it is made up to be more complicated and with more details. I’m not trivializing it or saying that it is extremely simple.
But I think there is a way where even conceptually and you know, have very strong intuition for item response theory. And as a product designer, as an instructional designer, you may be able to leverage that insight.
What is IRT?
Let’s quickly move ahead. And let’s understand what are the applications of IRT. So, test standardization GRE we already spoke about it, consumer sentiment, judgment anywhere where there is some sort of unlocking of latent, hidden sentiment is required.
IRT may be useful. So, in questionnaire design, you’re going to come across those five point scales, seven point scales, nine point scales, positive scales and lots of those things. But how do you analyze that data meaningfully
I think unlocking the inside from some of those things that’s where the item response theory comes into play. So, it has like a lot of layers built into it.
And finally, I think the you know, leadership assessment research selection process, I think this is somewhere if you look at tasks, which is more the executive search firms, they may be using something similar to this for at least shortlisting or some early stages of evaluating some talent.
The number one thing that I put on this particular slide is the programming ability and that is something that is of enormous interest to me and something that I’ve seen a play. So, it does not have a 100% proven utility over here.
Item response theory being used to assess evaluate benchmark programming abilities are not seen very strongly frequently, but it is something that can be done there is some amount of research and some amount of work that has already been done in literature globally, and something that we have also seen, you know, doing it.
So when we come to be able to discriminate a good programmer from a bad programmer, and I’m just trying to make some very broad generalizations, the difficulty comes into play.
So I think marrying that difficulty and good programming and bad programmer, if we were to just take these two things, if you take two different tests, and one test has 10 questions, but all easy and the second test has 10 questions all difficult.
Someone who gets 10 on 10 in the first test with all these equations out there, and someone who gets four on 10 on a questionnaire which has all difficult questions, how do you compare those two candidates?
Maybe IRT can give us some direction in that particular case. The ability to discriminate and factoring in the right kind of difficulty measuring the difference quantifying it and then throwing out some standardized scores.
Maybe that is what IRT really does over here. So these are some of the applications of it. And I’m sure that you may be if you’re thinking from a learning mindset or learning outcome mindset, this is something that is of total interest to you.
Why should the Learner Outcome Engineer care?
Why should you as a product manager, as a learning outcome engineer, as an instructional designer care about it? So I think at the heart of the measurement framework, when you are having students in the system, whether it is your classroom, it is an online, classroom environment cohort batch; however you define it, or it is a cohort of one kind of thing. A measurement framework is at the heart of being able to give some sort of meaningful feedback.
So to plan any kind of instruction and the following intervention, you will need to have a very strong robust measurement framework. So how do you unlock some of these insights? How do you get the best Out of the measurement framework, questions, quizzes, assessments that you put in there and plan some of these interventions.
This is somewhere this will be very useful. A learner is very likely to trust you if the learner finds that this particular software system online product tools test gives me a very valued genuine kind of feedback. If the tool comes back and tells you that this is where you messed up, this is where you went wrong, and exactly pins pinpoints that out for you.
I think it’s a really valuable insight, when to show that thing, how to show that thing. A lot of UI UX experience considerations come into that, but that insight person is extremely valuable. So this is something again, can be unlocked out of this particular part. I would like to bring out and I’ll share some of these things. You know, going more into the measurement framework standardization, measuring skills.
So NASSCOM has something called qualification packs, and we’ll have a look at it. And something called a medicine project. This is a global kind of a thing. So a lot of people who do a lot of these things for different fields and different competencies and different areas, so we’ll have a look at it.
And finally, guys, I think if you’re a product manager, revenue is somewhere you consider learning outcomes. Yes. But learner outcomes engineer has a twofold kind of objective where you want to achieve the learning outcomes and also want to make sure the revenue the learner also pays something tangible, to the organization for the product that he or she is using. We’ll just quickly jump into what this NASSCOM qualification pack is and additional projects are before that.
A Limitation
I just want to put one last caveat over here, that item response theory is not a solution for everything. It is not something that can be applied blindly or has to be put in. It’s not like that topping. It’s not that tastemaker there’s something that needs to be first understood so that it can be used intelligently.
So that’s one particular thing, human evaluation and feedback that a lot of these online platforms where people code, they submit their code or people build projects, and they submit their projects. And then the human evaluator at the other end, with a certain amount of delay comes back to you with qualitative slash quantitative feedback.
The instructor in this particular case is using a document called a rubric, which is essentially just a document that is going to help the instructor please evaluate this assessment on these 12 points. It’s a very well documented rubric and maybe I think of one of these videos and share what a rubric typically looks like.
So that’s what may also solve your business problems that many times you don’t necessarily always have to get into item response theory, quantify it, measure it out, report it down.
This may solve your problem. Just Enough, one playlist that I want to share with you for all those who are slightly mathematically inclined, I think you should definitely get started with this particular video, which is the conceptual introduction to item response theory. I’ve gone through all these six videos, I understand, it’s close to 60 minutes worth of content.
And it’s, it’s good. It’s really very well explained the examples the, you know, shoulder ability that the author speaks about in one of the first and second videos where he just drives home the point you know, and it starts typically informal point where, you know, what is the classical theory item response theory different and gets slightly mathematical at times, but I think that’s acceptable. It’s a very well done kind of a video
I would recommend that you go and have a look at these six videos. 60 minutes, I think it will definitely add a lot of value to what you’re doing. So there are a lot of things that are coming on the item response theory.
From my side, we will do a broad literature review of pair item response theory at play in use at this point in time in the industry, we’re going to do some sort of a mathematical deep dive where I’m going to try and help you connect something that we learned in sixth standard, and build it from there from your sixth grade to where you are.
We will do a working session here. Let’s see how that thing goes. Machine learning and IRT, you know, you will come across things like DIRT and some interesting, fascinating applications that are being built, some interesting work that is being done in this particular space.
And finally, I think we’re going to have a working session where we’re going to really talk. I have understood what it is. But how do you really apply it? I think, can we move from item response theory to item response, you know, application, so to speak, and that’s like a very bad analogy, but I think how do you apply what you learn?
That’s probably the thing. I probably see like a couple of sessions. Just coming on this so I’ll keep you guys definitely posted on this.
Before that I just want to quickly jump onto this. So this is a website, which is ssenasscom.com. So, NASSCOM has these things called as qualification packs.
And what qualification pack essentially, is that for a particular role for a particular role, what is it? What skills should a person have, and then objectively made a list of that particular thing.
So, let’s just quickly have a look at what a qualification pack typically looks like. And the reason why it is important is that, you know, if you’re able to pinpoint that for a person who is in this particular sector or this particular sub sector, what are the skills that are that we are looking at and if we can objectively you know, put them out.
Fairly, you know, in all kinds of documents but it kind of comes to this KA1, KA2, KA3, etc you need to know you should be able to do this, and performance criteria and all those kinds of things. So maybe I’ll just quickly take you through another document, which is, you know, the Edison project.
But a recent project is I think these guys in the year 2015, they started about formalizing the data science profession and building out you know, what should be the skills required to get into this particular profession. I don’t really have any comment on what they wanted to achieve, or what they did end up achieving. I have no first hand information on that.
But what I would really like to point out is that they have put together some really interesting documents, and one of them is the DS-Bok. That is the book of knowledge for data sciences. And in this particular book, on one of the pages is this particular thing, a detailed definition of the DS-Bok and suggested knowledge units.
I just want to pinpoint your attention to a specific place: Knowledge area groups, there are knowledge areas within the knowledge area groups, there are knowledge units. And within the knowledge units, there might be even more subsections.
So what I really want to, you know, bring your attention to is the three level hierarchy, knowledge area, group, knowledge area and knowledge unit, the way they have broken down a subject, the taxonomy that they have tried to establish the hierarchy that they have tried to establish the way they are documenting that this is what the body of knowledge is not this particular subject is, is something that I want you to take over here.
And this is at the heart of what is a measurement framework, any measurement framework is going to need some sort of, you know, how are you going to count what you want to count. So, this is something that is a very well thought to kind of, at least, it’s a good start. I think you should just go to this particular site, edisonproject.eu and discover more about what
They have to offer there are multiple such books with respect to the competency framework with respect to the book of knowledge. And you’ll find some value at least skimming through some of these documents.
So these are the things I just wanted to quickly pinpoint to you in the context of, you know, the item response theory. And just coming back, why item response theory and why some of these things, the simple reason being that item response theory, built on top is used in conjunction intelligently with a very strong measurement framework. With a very well defined measurement framework.
Let me not say strong or weak but a very well defined measurement framework might be your iterated path to build something meaningful with respect to a strong, meaningful, valuable product, valuable experience for your learner.
So that’s pretty much it what we wanted to offer you with respect to what is this item response theory one of the simplest introductions, we just wanted this to be an introduction to item response theory.
And I look forward to hearing from you. I’ll be posting some of these videos on my blogs. And a series too with part two and three of this particular blog. That’s coming up. Thank you so much.
Stay tuned to humans of EdTech and I’ll be sharing my thoughts on the world of EdTech and what’s happening, what’s cutting edge and what to expect in time.
Thank you so much, bye.
Photo by Lukas from Pexels
Join to get sneak peek into what's happening
I write about books, experiences, product, UX, EdTech, early stage growth, validation – mostly tech. Subscribe if these topics interest you. Once every 15 days emailer. I promise – No spam. (I am known for it otherwise) 😉
