User-oriented metrics for search engine deterministic sort orders

A recent paper proposes the "buying power"(bp) metric for assessing the quality of the product rankings generated by e-commerce sites such as Amazon and eBay. Focusing on the "ordered by price'' type of product listing that is often viewed after a keyword search, bp is offered as a way of differentiating between helpful rankings (high bp scores) and unhelpful rankings (low bp scores), with those bp scores intended to reflect both the quality of the product match and also the relative pricing of the items that are listed. In this paper we adopt a user-centric viewpoint from which to evaluate the merits of bp as a scoring mechanism for product rankings, and provide an example that shows bp acting in opposition to likely user reactions. We then describe an alternative product ranking effectiveness metric, price biased gain (PBG), arguing that since it embeds a more plausible user model, it is more likely to reflect the opinions of the user viewing any given product ranking. We give a number of scenarios and motivating examples in support of our alternative proposal, and also discuss its limitations.