We discover the limitation of existing methods for searching paintings interactively on how well the query is represented by the user, and propose a simple, yet effective way to search the painting by exploiting the color to express human visual memory.