Can I do the search testing of engine results by myself?

Actually, the question of search testing engines is part of the information retrieval (IR) problem. I will list just a few approaches how they can be evaluated/tested. Traditional IR approach One way in traditional IR is to prepare test setup, including:

Candidate queries Dataset of indexed pages Expected result set (included expected ranking) for each query Next you obtain actual results by running your search engine over candidate queries and compare actual results with expected results using one of information retrieval metrics, e.g. precision/recall or nDCG. That would give you a quantitative answer how bad or good your search engine is with respect to expected result set (requirement).

Obviously, the answer will be context-specific. It will particularly depend on choice of candidate queries, their intention and wording, the dataset of indexed pages, and the way expected results were judged. So what was the context?

Goal-driven/Interactive IR approach Another approach, maybe more practical, would be to involve potential users and let them assess the quality of results for a given query. They could bring their own queries or you could define queries for them. Those could be a few people or a whole community, from which you could get more implicit feedback about results quality (e.g., results that are clicked are expected ones).

Problem isolation Finally, the problem might be not to evaluate the overall quality of the search engine, but only narrow down the reason for low ranking of this particular query result. So, I would follow the problem you got: You were told that a search engine result by target query does not satisfy a requirement by its position.