How to use Color Histograms for Image Retrieval

00:00:00.000 | Browsing, searching and retrieving images has never been a particularly easy thing to do.

00:00:06.800 | Traditionally, many technologies relied on manually annotated metadata and searching

00:00:16.000 | through that using the more established or better established text retrieval techniques.

00:00:23.840 | This approach works when you have a data set that has high quality annotation, but

00:00:31.120 | this is really not the case for most larger data sets. That means that any large image data set

00:00:40.480 | has to rely on content-based image retrieval. Search with content-based image retrieval relies

00:00:48.240 | on searching through the content of the image rather than relying on any metadata. Content can

00:00:54.960 | be color, shapes or textures, or with some of the latest advances in machine learning, deep learning,

00:01:02.160 | we're actually able to search based more on the meaning of an image, like the human meaning,

00:01:08.000 | rather than even the textures or edges in the image or so on. But today, we're not going to

00:01:15.040 | be looking at that, we're going to be looking at one of the earliest methods for image retrieval,

00:01:20.480 | which is color histograms. Color histograms represent one of the earliest content-based

00:01:28.320 | image retrieval techniques, and they allow us to search through images and retrieve similar

00:01:33.680 | images based on their color profiles. And we can see an example of what we're going to learn about

00:01:39.760 | here. So we have this query image, it's just a city, you can see it's very blue and orange,

00:01:45.280 | and to the right here, we have this, it was basically, this is what the color histogram is.

00:01:51.520 | Now, when we actually build our color histograms and search with them, we are going to convert

00:01:57.920 | them into a single vector, so an image embedding. And with that image embedding, we're going to

00:02:04.400 | return these five images. Now, this is using a very small dataset. We have, I think it is literally

00:02:11.680 | 21 images, it's tiny. So with that in mind, this is returning pretty relevant images when we speak

00:02:20.160 | about the color content of these images. They all seem to have this similar sort of aesthetic,

00:02:26.560 | a lot of blues and oranges. So this is pretty much what we're going to do. And underneath these

00:02:31.600 | images, you can actually see the color profiles as well. So the one to the right is actually just

00:02:37.120 | a duplicate image, it's the same image. But then the other ones, we can see that they all have this

00:02:42.480 | kind of, or some of them have this peak in the red, and then they're pretty flat with the rest

00:02:48.400 | of the colors. So this example demonstrates the core idea of color histograms. That is, we take

00:02:55.040 | an image, translate it into the histograms that you saw, which we then translate into a image

00:03:00.800 | embedding vector, and then use our color profile embedding to retrieve other similar images based

00:03:08.480 | on their color profiles. Now, there are many pros and cons to this technique. I mean, as you'd expect,

00:03:14.320 | it's one of the earliest techniques for image retrieval, but there are a lot of benefits. And

00:03:20.640 | we'll discuss a lot of those as we go through this video and just understand how to actually

00:03:26.960 | implement a color histogram. It'll become quite clear what the pros and cons are, if you haven't

00:03:31.600 | figured a few of those out already. Now, we're going to work through two notebooks. The first

00:03:36.880 | one's just going to show us how we actually build color histograms, so we can actually see step by

00:03:41.120 | step the actual process. And the second one is where we'll implement the search component. Now,

00:03:47.200 | you can find links to both of these notebooks in the description of this video. So if you want to

00:03:53.120 | follow along, go ahead, open those, and you'll be able to go through the code live. So a few things

00:03:59.680 | that we need to install. So just pip install, we have OpenCV library, NumPy, and datasets. Now,

00:04:06.560 | OpenCV is just a public computer vision library. NumPy, I'm sure you probably know what it is.

00:04:12.960 | It's just focused on numerical operations for arrays. And datasets is HuggingFace datasets.

00:04:21.040 | HuggingFace is like an NLP library, and datasets is their way of allowing people to store datasets

00:04:30.320 | and download them super easily. So for us, we are going to use that to get this image set here. Now,

00:04:38.880 | you can use your own images if you like. You don't have to do this one, but this is what I'm going

00:04:43.920 | to go with. And this is, like I said, it's, I think, 21 images. It's really not that many,

00:04:51.120 | but it's all we need for this example. So you can load the dataset. So from datasets, load dataset,

00:04:58.080 | and we have this pinecone image set. If you'd like to see where this dataset is, you can go

00:05:04.640 | to HuggingFace.co/datasets, and then you just type in the dataset name that you see here,

00:05:10.640 | pinecone/image set. And it will take you to where, yeah, this website here, which is where

00:05:16.480 | this dataset is actually hosted. Now inside, actually, let me run this. So inside here,

00:05:25.520 | we have this image bytes feature, and this is just a base 64 encoded representation

00:05:33.840 | of our image bytes. So when we download these, we need to decode them, again, using base 64.

00:05:43.200 | So we do that here. So we're just going to create this processing function,

00:05:47.440 | and we're going to decode everything. I don't think there's really much to go through there,

00:05:54.880 | but from that, we'll get these images. Okay. And we can actually check. So let's just see

00:06:02.320 | how many images there are. 21. Okay. And this should align to appear as always number of rows

00:06:09.760 | 21. Okay. Cool. And we can display the images now with matplotlib. And you see we get this image

00:06:20.560 | with these three dogs, but they're pretty blue. That's because we have loaded them,

00:06:28.480 | or we've decoded them using the OpenCV library. And OpenCV library, it reads images using the

00:06:37.120 | color channels, blue, green, and red in that order. And I don't know why they do that. It's

00:06:43.040 | kind of the opposite way of what most things do, which is red, green, and blue, or RGB.

00:06:48.000 | So what we need to do is actually flip those color channels if we'd like to see the true color

00:06:53.680 | version of this image. So let me show you. So at the moment, this is a shape of that image. So

00:07:03.760 | this image is actually an array. These two here are like the actual pixel values. So you can see

00:07:11.280 | two, five, six, zero at the bottom here. And then you can see one, six, zero, zero over here.

00:07:19.120 | Okay. So it's the Y, Y axis or the height of the image, and this is the width of the image.

00:07:25.600 | And then this three, these are the color channels. So it's the blue, green, and red color channels.

00:07:32.720 | And here are the color values. So blue, green, and red for the very top left pixel,

00:07:45.360 | right up there in the corner. Okay. So we can see these values, by the way, so they go from

00:07:53.200 | zero, which is no color up to 255, which is full color. And we'll have a look at that in a moment.

00:08:00.000 | I'll show you what I mean. So first let's just flip those color channels. So you do NP flip.

00:08:06.720 | We are going to flip the whole array, but we are going to flip it from axis two, which is a color

00:08:14.640 | channel axis. So let's do that. And you can see the shape is exactly the same because all we've

00:08:21.600 | done is flipped it, but these, the pixel values here, so it was blue, green, red is now red,

00:08:29.680 | green, blue. So now we can visualize that and we actually get the actual picture, which is these

00:08:36.560 | three dobs that are very much not blue. So first we're going to go through building a histogram

00:08:42.240 | the slow way, just so we can understand exactly what is actually going on. So let's take a look

00:08:47.760 | at image zero again. So the three dobs, and we're going to have a look at pixel zero. Okay. So we

00:08:54.240 | have those, those values. Again, this is the other way around. So it's blue, green, red.

00:08:59.760 | Now each pixel, like I said, has those blue, green, red activation values from zero, no color

00:09:08.000 | to two, five, five max color. So if you had, let's say these were zero, zero, and zero,

00:09:16.080 | there's no color or there. That means we would get just black. Okay. Cause there's no color.

00:09:23.040 | If we had two, five, five, two, five, five, two, five, five, then we have white, which is just

00:09:29.920 | all the color you can possibly have in one. Okay. And in between them, you have everything.

00:09:37.760 | So here's a few examples. We have blue, green, red, and a few other things. And I'm just going

00:09:43.280 | to plot those in this array here. And I'll show you. Okay. So we can see we have blue, green,

00:09:53.120 | red, and so on and so on. Okay. And you can see all I'm doing is swapping the two, five, fives

00:09:59.680 | for the orange one. We've got like half of the green color in there as well, but that's pretty

00:10:07.760 | much all we're doing. So every single color can be represented by these values. So back to the

00:10:15.760 | previous values, we have that, the top left corner pixel of that picture of three doves,

00:10:23.840 | we have blue, green, and red. So you can think, okay, blue, green, and red, that means it's going

00:10:29.040 | to be sort of a greeny blue color, which is going to be quite neutral because you have all of those

00:10:35.840 | colors kind of in the middle between zero and two, five, five. So we can see that here. So this top

00:10:44.400 | left block here is actually that pixel, right? Because here, I'm just displaying the three by

00:10:54.400 | three pixels in the top left corner of the image. Okay. And this is a true color image, by the way,

00:11:00.720 | that's why it's RGB image. So that's the one we flipped. Now what we are going to want to do when

00:11:09.520 | we're comparing these images is actually, we don't want an array, we want a vector. So I'm going to

00:11:17.680 | reshape this. Okay. And now we have these 409, or four and a little bit million values, which is

00:11:31.840 | just all of the rows in our image, or sorry, yeah, all the rows concatenated together. Okay. That's

00:11:41.440 | what this reshape is doing. Now, if we plot that again, we can still see that those top three

00:11:48.640 | values are the same. Okay. But there's no more rows underneath those pixels anymore. It's just a

00:11:55.520 | single row. Okay. Now, okay, we have this one row, but it's still actually an array because we have

00:12:02.400 | the three color channels. So we need to also extract those out. Okay. And now we literally

00:12:10.240 | just have three vectors that represent our image. And we can visualize each of these with a

00:12:16.000 | histogram. So let's do that. And this is the color profile, the red, green, and blue color profile of

00:12:26.880 | that image of the three dobs. So what we have on the X axis here is the color activation value from

00:12:34.880 | zero up to 255. And then what we have on the Y axis here is the count of the number of pixels

00:12:42.080 | that have that particular value. So, I mean, we can see this is a pretty neutral color. Okay.

00:12:48.160 | Most of the values, and this is probably the case for most images as well. Most of the values are

00:12:53.680 | kind of in the middle point. So it's pretty neutral. You can see that there's a lot of

00:13:00.800 | pixels here that don't have any blue in whatsoever. But beyond that, I don't think there's as much to

00:13:07.280 | note from there. Okay. So let's put everything we've done so far into a single function so that

00:13:14.800 | we can replicate these charts for a few images. So what did we do before? We had our image.

00:13:21.760 | We can also change the number of bins that we use. So you see up here where we've got

00:13:26.800 | an individual bin for every single color activation value. We can push those together

00:13:32.400 | so that we have less bins and we are going to do that later on because it doesn't really affect

00:13:37.200 | the retrieval performance that much unless you go really low. So first we're going to convert

00:13:44.000 | to a true color image. So from BGR to RGB, I'm just going to show the image so we can see what's

00:13:50.160 | actually happening whenever we call this function. Convert it into a vector with the three channels.

00:13:56.480 | And then what we're going to do is, so here I'm breaking the values or dividing the values by the

00:14:07.840 | number of bins and then converting them back to integer values. This is basically just a really

00:14:13.520 | quick way of creating the bins. So if I divided this by two, for example, so I had two bins here

00:14:23.120 | we'd get 128 in this division parameter. And we'd divide everything by 128. So we're kind of

00:14:32.880 | pushing everything together, all these pixel values into discrete categories or bins.

00:14:43.680 | And then we want to get the red, green, blue channels. And then we plot it. So I'm going to

00:14:48.400 | run that and we're just going to try it on a few images. So we'll start with this one.

00:14:52.720 | So we have this image of a city and this is, I think, where you can see color histograms are a

00:15:00.480 | bit more interesting than the last one. So it's a very blue image. And we can see that here,

00:15:06.080 | like it is super blue. And then we look at the histograms and the blue histogram, there's a lot

00:15:11.040 | of high values for blue pixels over here. Now, what you can also do, which I mentioned before,

00:15:19.280 | is use the bins parameter. So we just want, let's say we want 64 bins here.

00:15:24.160 | And you can see that we have these sort of bars now where you can really see those before. That's

00:15:32.240 | because we have 64 values and we can reduce that even more. Let's say we wanted to go really low

00:15:37.760 | and we want to go to two. Okay. And we just get these two bins now. Okay. And you can still see

00:15:45.920 | even from these two bins, it's a very blue image, but of course that's a little bit too much. So

00:15:51.760 | we'll stick with sort of values between 32, 64, because you can actually see what's going on with

00:15:58.400 | that. Now, if you have a look at this image, we can see there's very little blue in this image.

00:16:05.920 | It's a lot of green, a little bit of red as well. If you take a look, we can see that almost all

00:16:15.200 | of the blue values are pushed right to the left, which means there's very few high value blue

00:16:21.360 | pixels in this image. We can do this again here. We see, okay, again, it's very green. So don't

00:16:30.000 | really get those blues and we can keep going through all of that. See a few more. And this

00:16:37.280 | one, this one's also interesting because it's a very color specific image. It's a lot of orange

00:16:43.440 | there. And you can kind of see that because it just got these really big spikes in particular

00:16:49.680 | areas of your histograms. So, well, we can see that represented by the histograms.

00:16:56.240 | Now that's a slow way of building histograms. Like I said before, we've gone through that to

00:17:04.160 | understand exactly what is going on, but it's not the most efficient way of doing it because

00:17:09.200 | there are already functions for creating histograms built into the OpenCV library.

00:17:15.440 | So let's have a look at how we would use that. So we have this CV2, we've imported the CV library

00:17:24.160 | earlier, just here. So just import CV2. And then what we're going to do is we create, we use this

00:17:32.560 | calc_hist function. We pass in an image, we pass in the color channel. We can only do this one color

00:17:40.160 | channel at a time. Whether you want to mask anything, I'll explain that in a minute and so on.

00:17:45.440 | I'll explain the rest in a moment as well. So we'll run that and you see that we get this 64 by 1

00:17:52.640 | shape. And this is actually our histogram. So we don't even need to use, before we're using

00:17:58.000 | matplotlib, the histogram function there. Now we don't even need to use that. We can just plot this

00:18:03.680 | directly. Now there are a few things in here. So we have this calc_hist function. What are all these

00:18:10.560 | values? So we have images, channels, mask_hist_size and ranges. What does that mean exactly?

00:18:18.160 | So the images. So this is a list of CV2 loaded images with the channels blue, green and red.

00:18:27.600 | So if you look up here, that's why I've taken the red histogram from the third channel or position

00:18:34.800 | two and green obviously in one and blue in zero. It can load multiple images. So that's why we have

00:18:42.080 | put that single image inside the square brackets, where is it? Here. And then we have channels. So

00:18:52.480 | this is what channels you want to create your histogram for. I'm wanting to extract one at a

00:18:58.720 | time here. So I've said, okay, I want the red channel or the channel in position two and so on.

00:19:06.400 | Mask is another image or array which just consists of zeros and ones. Now that allows us to mask a

00:19:15.920 | part of the image if we would like to. So imagine you had half of the images zeros, half the image

00:19:23.200 | is ones. It means that you would literally remove half of the image when you add that mask to this.

00:19:30.240 | Okay. Imagine you multiply all the values in your image by the zero ones, the ones that become zero

00:19:37.040 | there, you can't see them anymore because their color activations are zero. Okay. So they just

00:19:41.840 | become black. That's how it works. And then bins is the number of bins that we'd like to add in

00:19:49.440 | there like we did before. And then the histogram range is the range of color values that we would

00:19:55.920 | expect because we're using RGB, we expect zero to 255. So rewrite zero to 256 because the top value

00:20:06.240 | is not inclusive. So that is not included. So actually just goes up to 255. Okay. And let's

00:20:12.240 | have a look at what we get from that. So like I said, we don't need to use the histogram plot from

00:20:17.360 | that plot lib anymore. We can just plot it directly and yeah, we get this. Okay. So I think this is,

00:20:25.040 | is it the last image? Yeah. So it's this image here. You can see that we get the same thing.

00:20:32.080 | So that's the end of the building histograms notebook. Let's have a look at how we actually

00:20:38.080 | create our embeddings with this and then how we actually search using these color histograms.

00:20:45.120 | Okay. So we're now in this search histogram notebook and what we're going to do is create

00:20:50.080 | a function to basically do everything we've done so far. So using the CV2CalcHistForth function,

00:20:57.920 | we're going to use that. And then we're going to actually concatenate the red, green, and blue

00:21:03.520 | channels into a single vector. So you've seen this before. We have red, green, and blue.

00:21:09.840 | Then we concatenate red, green, and blue together along axis zero. And then we reshape. So this is

00:21:16.800 | a minus one. That's just to remove, when you run this, there's always like an extra dimension in

00:21:22.560 | there. So we're just removing that extra dimension. Now, if we run that, we should see that we get

00:21:28.880 | this. So we have a 96 dimension vector. Now, why is it 96? So we have, we just set the default

00:21:37.760 | number of bins to 32 at the top here. So that means we'll get 32 values for the red channel,

00:21:42.560 | for the green value and the blue channel, and then we concatenate them all together. So in the end,

00:21:47.360 | we have all those 32s together, three of them. So we get a vector of dimension 96.

00:21:53.440 | So if you also imagine this, so let me visualize it even. So we run this. You can see here,

00:22:02.480 | we have the vector, the values from up to 32 here from 32 to 64, and from 64 to 96. And these are

00:22:12.800 | red, green, and blue. And we get this. It's the same thing as what we saw before, but we just

00:22:18.640 | separated them all into a vector rather than a array with three color channels. Now let's go

00:22:26.800 | through and we'll just use this loop to do that for all of our images. So we're going to create

00:22:30.880 | all of these image vectors and we can compare vectors with Euclidean distance, although I found

00:22:38.800 | this didn't work very well, at least not compared to cosine similarity, which we calculated like

00:22:45.920 | this. So what we're going to do is use cosine similarity to find the most similar matches for

00:22:54.800 | each image, and we're going to pull this within a function to keep everything clean. During

00:23:01.040 | visualization, we're also going to use the deflect arrays, so the true color arrays.

00:23:06.080 | So I can actually see what they are a little bit better. So I'm going to run that. If I go through

00:23:14.960 | here, all we're doing, we're getting the query vector, which is marked by this index value here.

00:23:21.040 | So for example, those three dogs, they were position zero within our images. So if I wanted

00:23:26.880 | to use that as our query vector, I would just pass zero to this function here, and that will

00:23:33.200 | retrieve the query vector from the image vectors we've already created. Now I'm going to go through

00:23:38.160 | and I'm going to calculate the distance between that query vector and that query image and every

00:23:44.080 | other vector within our image vectors. That also includes the same image. I mean, it's not hard to

00:23:51.920 | remove that from here, but just for the sake of simplicity, I've kept it in. Plus it tells us if

00:23:57.040 | this is actually working, because we should always return that one as the most similar image.

00:24:02.720 | And then we're using this NumPy arg partition function, which is just based on the distances

00:24:12.080 | we calculate, it's going to retrieve those indexes. And then we can use those indexes

00:24:19.280 | to retrieve the images, the actual images themselves that are the most similar. So if

00:24:26.960 | we just run that using the dog image, we see that we returned dog image first, because that's the,

00:24:32.560 | oh, sorry, this is reversed. So this is actually the first most similar image. And

00:24:39.920 | that is the dog image itself. And then we have these other ones. We didn't know what those are.

00:24:46.000 | So let's go ahead. I'm going to write another function. This is not so important, but this

00:24:51.120 | just going to help us visualize these results. So I'm just, there's a lot of NumPy here. We don't

00:24:57.680 | really need to go through this, but you can go through this code if you want to. So run that,

00:25:04.480 | and then I am going to get some results, same as what we saw before. And I just want to visualize

00:25:10.560 | those. So here I'm actually doing it for another image, so we can ignore that bit there. So for

00:25:17.280 | image six. Okay, cool. So we have, this is the image that we're using as our query. And you can

00:25:26.160 | see, this is what I showed you at the start. So it's the city image. And we have these, this is

00:25:32.560 | the color histogram for it. And then these are the images that it's returning. So very similar in

00:25:39.520 | terms of the color scheme for each of these. Let's try another one. Okay. So this one, I mean,

00:25:48.160 | it's pretty obvious what this should return, probably another image. As we can see, like that

00:25:53.280 | one. So we have this yellow background with the dog, and then we have the yellow background with

00:25:57.760 | a cat. Okay. And then there's some other images. These ones are not really that similar, but there

00:26:03.200 | aren't that many images in this dataset. So that's why there aren't any others that are more similar.

00:26:10.640 | And this one, I think is really good because we can really see that these two color histograms

00:26:20.000 | are very aligned, like they're very similar. And if we want to just have a look at, so at

00:26:26.480 | the moment we're using all of the, we're using a default number of bins. I'm not sure what the

00:26:32.720 | value is. Anyway, we can modify the number of bins. So we'll go 96. Let's see what we get.

00:26:40.480 | Okay. And we get this. So now we're modifying the bins. We're still returning the same images. So

00:26:49.040 | there isn't, I think one of these might be slightly different.

00:26:52.240 | Okay. So this one here, actually reducing the number of bins here did damage the quality of

00:27:06.320 | this retrieval. So in this case, we still have that relevant image, but it's, we have these

00:27:12.880 | other images that are actually being placed before it, which is surprising, but it's just

00:27:19.280 | one of the limitations of this technique is it's not particularly robust. Now this is,

00:27:27.920 | this is how we would actually search using these color histograms. Now I think it's probably quite

00:27:34.960 | clear what some limitations of this are. This retrieval technique isn't perfect. And I think

00:27:41.600 | these results highlight some of those drawbacks. So the key limitation here is like, okay, we come

00:27:48.480 | up here and we can see, yeah, we were getting, we're returning this image of a cat with a yellow

00:27:54.560 | background. But what if for your use case, you're, you're not actually looking for the similar color.

00:28:01.680 | You're actually looking more for similar content. So you'd actually want to return, okay, this is a

00:28:07.440 | dog. I want to return another dog. So probably this image over here, in that case, this is a

00:28:13.440 | really bad technique because it's basing everything purely on the color. It's not even looking, it's

00:28:18.880 | not looking at textures and not looking at the edges of an image. It's not looking at what is

00:28:24.480 | inside the image. It's just looking at the color profile of each image. So it's very limited with

00:28:30.800 | that. Now there was further work on color histograms to improve some of these. So for example,

00:28:37.440 | different techniques that consider the texture of images, they consider the edges within the images

00:28:44.960 | and a few other things. But still all of these things, they're not going to get you as far as

00:28:51.920 | some of the more recent deep learning methods that allow you to actually consider what is inside the

00:28:57.440 | image from a very sort of human perspective. But that being said, if you just want to return images

00:29:07.040 | or retrieve images that have a similar sort of aesthetic to a particular image that you're

00:29:11.440 | searching with, say even a similar color profile, like we saw with the earlier results up here,

00:29:18.080 | we are returning pictures that kind of look like the first image, like the query image. So that's

00:29:26.800 | one pro to using this technique. Another one is that it's incredibly easy to implement. We've just

00:29:35.040 | done it. And in reality, we don't even need to go through that sort of slow building a histogram

00:29:40.400 | part. We can just do it super quickly using OpenCV. And another key benefit is the results

00:29:49.120 | are very interpretable. So with a lot of deep learning methods, you have a black box, you put

00:29:57.200 | in some data, you return some results. Why did you get those results? A lot of the time you don't

00:30:02.800 | really know. With this, you know exactly why you're returning a particular result. You can see

00:30:08.640 | that the color profile is very similar to the one, the particular color profile that you're querying

00:30:13.600 | with. So we understand why that's actually happening here. It's not a black box like neural

00:30:18.880 | networks. So with all that in mind, this approach to image embedding and retrieving images is great

00:30:28.560 | if your use case is looking at the aesthetics or the color profile of images. If you want something

00:30:36.400 | more advanced, of course, you're going to want to go for something else. Now that's it for this

00:30:41.600 | video. I hope this sort of introduction to 1D earlier, image embedding techniques and methods

00:30:51.600 | for content-based image retrieval has been interesting. But for now, we're going to leave

00:30:56.480 | it there. So thank you very much for watching and I will see you again in the next one. Bye.

How to use Color Histograms for Image Retrieval

Chapters