Find similar call with "matchPerson" mode is not actually matching person anymore.

Question

Find similar call with "matchPerson" mode is not actually matching person anymore.

Roshan Giri 0

I used to get great results with Find SImilar call with "matchPerson" mode.

It was as good as identify call on Person Group, and I need not check the confidence of result as well, as the internal threshold for this mode was enough.

But on some instances I have found it to return extremely poor response with "matchPerson" mode returning completely different person, of different gender, different age and also different race. I would have understood minor errors, but the results were very shocking. Do we now need to filter out based on condifence even with "matchPerson" mode, and if so, why do we even need matchPerson, would not matchFace be enough.

Or has anything changed with how these work recently?

2 answers

Your answer

Answer 1

Hi,

I'm pretty shocking to hear. when a 'matchPerson' mode returns someone of a completely different gender and race, something is definitely off. that should not be happening...

first, let's clarify the modes. 'matchFace' just looks for visually similar faces, ignoring identity. 'matchPerson' is supposed to be smarter. it uses a person model to understand the actual identity, filtering out faces that might look similar but belong to different people. it's meant to have that internal threshold you trusted.

the fact that it's failing so badly suggests something might have changed on the backend. microsoft occasionally updates its underlying ai models to improve accuracy, but sometimes these updates can introduce regressions for specific use cases.

your immediate fix is to absolutely start checking the confidence score. even with 'matchPerson', you should never treat the top result as absolute without verifying its confidence. set a threshold for your application. what's acceptable? 0.7? 0.8? you will need to test to find the right balance for your data.

to troubleshoot, try running the same face against your person group using the 'identify' call. does it also return the wrong person with high confidence? if yes, the issue might be with the quality of the faces registered in your person group. if 'identify' works correctly but 'find similar' does not, then the issue is likely isolated to the 'find similar' api.

finally, please report this behavior to azure support. provide them with example request ids and the specific faces that are causing the poor matches. they can investigate if there's a service side issue or a bug in the current model. your feedback could help them fix it for everyone.

really hope you get back to those great results soon. that kind of accuracy is why we use these services.

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer

https://ctrlaltdel.blog/

Roshan Giri 0 Reputation points

2025-09-18T16:51:37.6233333+00:00

Thank you, yes I am raising a support ticket.
It is concerning as our solution made sure of using high quality faces based on face attributes returned on the detect call.

And yes, will add a threshold for the time being on our side while we wait for answers,

Answer 2

SRILAKSHMI C 10,805 Microsoft External Staff Moderator

Hello Roshan Giri,

Welcome to Microsoft Q&A and thank you for reaching out.

Let me clarify how the Find Similar API works with the matchPerson mode, and why you may be seeing different results than before.

The matchPerson option is designed to return results that the service determines to be the same individual as the query face, based on an internal threshold. In the past, this threshold often allowed customers to rely on the output without additional filtering, as it was comparable in accuracy to using the Identify API with a Person Group.

That said, there are scenarios where matchPerson may return false positives for example, matching a different gender, age group, or even race. This can happen due to factors like poor image quality (lighting, pose, occlusion), mismatched or outdated recognition models, noisy candidate data, or updates to the underlying service that change how thresholds are applied.

Because of these factors, we now recommend that customers apply a confidence threshold to the results even when using matchPerson. Doing so ensures you maintain control over what qualifies as an acceptable match for your specific scenario. A common approach is to only accept matches with confidence above 0.7–0.8, but the exact value should be tuned to your business requirements.

It’s also important to remember the distinction between the two modes:

matchFace is best if you want visually similar faces, regardless of identity.

matchPerson is best if you want to detect the same person, but it should now be combined with confidence filtering to ensure accuracy.

Next steps we recommend:

Ensure that both query and candidate faces are processed using the same recognition model (e.g., recognition_04).

Apply a confidence filter to screen out false positives.

Periodically review your candidate dataset for quality and consistency.

Test with the Identify API in parallel to validate whether the issue is with your dataset or specific to Find Similar.

Please refer this Find Similar interface,

Find Similar Match Mode Enum.

I Hope this helps. Do let me know if you have any further queries.

If this answers your query, please do click Accept Answer and Yes for was this answer helpful.

Thank you!

Roshan Giri 0 Reputation points

2025-09-18T16:49:35.2933333+00:00

So, we are using matchPerson with the same recognition model.
We had experienced the issue with poor face data (ex blurry, angled, etc), so we ended up only adding the faces to facelist that are acceptable using the face attributes.
So, faces with attribute high on qualityForRecognitionare used, or if qualityForRecognition is medium, it needs to have blur < 88% and no mask and no nose and mouth covered flag. No other faces qualify for being added to faceList, or being used to as the face to find similar.

Hence I am a bit concerned about the result received.

On second note, if I need to add a threshold(seem like I will), I need to confirm that the results are always sorted by confidence with the best faces returned first?
SRILAKSHMI C 10,805 Reputation points Microsoft External Staff Moderator

2025-10-01T18:12:00.0533333+00:00

Hi Roshan Giri,

Thank you for the additional details it’s very helpful to know that you’re already applying strict quality criteria (e.g., qualityForRecognition, blur thresholds, occlusion flags, etc.). That does rule out most of the common causes for poor matchPerson results, so I completely understand your concern here.

To address your specific follow-up questions:

Yes, the results returned by the Find Similar API are always sorted by confidence in descending order, with the highest-confidence match listed first. This means that if you choose to apply an additional confidence filter on top of matchPerson, you can reliably use the first result (or top N results) as the most likely match candidates.

Even when face quality is carefully controlled, matchPerson results can occasionally degrade due to factors such as:

Subtle changes to recognition models or thresholding logic can shift how similarity is scored.

If candidate faces were enrolled long ago or under very different capture conditions, embeddings might differ significantly despite quality checks.

The model may be overfit on certain shared facial features even across demographics, causing false positives in rare cases.

Next steps:

Implement a confidence cutoff (for example, 0.75–0.85) and monitor distribution of match scores to fine-tune this threshold.

Consider comparing top N results rather than only the first one, especially if your application logic allows secondary verification.

Optionally run a parallel test with the Identify API on the same dataset. If Identify consistently outperforms matchPerson, it could indicate that the Find Similar scoring behavior has shifted for your data distribution.

We understand that matchPerson used to be highly reliable even without thresholding, and while the API design remains the same, we generally now recommend using confidence filtering as a best practice especially for mission-critical scenarios.

Thank you!
SRILAKSHMI C 10,805 Reputation points Microsoft External Staff Moderator

2025-10-02T10:54:48.31+00:00

Hi Roshan Giri,

Did you get any chance to review the above response. Do let me know if you have any further queries.

Thank you!

Share via

Find similar call with "matchPerson" mode is not actually matching person anymore.

2 answers

Your answer