QontoFAQ: A better Information Retrieval Benchmark [R]

Excerpt

Retrieval benchmarks sometimes feel benchmaxxed by models, so we wanted to find a way to tie it as close as possible to my objective: finding the article that answers a product question right . We worked on a new metric which seems more proportional to document relevance, and built up a benchmarking dataset to measure embedding models. Here is an article on the approach: https://medium.com/qonto-way/qontofaq-benchmarking-information-retrieval-acd89600ebe1 and the associated code: https://github.