CompsAPI evaluated its marketplace classifier, which tags sold-price listings as "item" (real product) or one of eight junk categories (lot, parts, accessory, box_only, mystery, replica, digital, other), using two independent human labels per listing reviewed for disagreements. Test sets comprised 600 listings from Reverb, Mercari, Poshmark and others (51 junk, 8.5% base rate) and 690 eBay listings from 20 unseen searches (277 junk, 40.1% base rate). On marketplaces, agreement reached 97.7%: 8 of 549 real items wrongly removed (1.5%) and 6 junk rows wrongly kept. On eBay, overall agreement was ~92%, decomposing into 24 of 413 real items wrongly removed (5.8%) and 30 of 277 junk rows wrongly kept (10.8%). A do-nothing baseline labeling everything "item" scores 91.5% on marketplaces and 59.9% on eBay, illustrating how base rates distort single accuracy figures. Error costs differ: missed junk skews median prices (10.8% on eBay), while removed real items hide deals (5.8%). Known weaknesses include accessories (35/41 caught), bundles occasionally misclassified (e.g., 2 of 23 Xbox Series X consoles removed), vague short titles, multipacks treated as lots, "body only" cameras as parts, real lots slipping through in foreign languages, and counterfeits sold as genuine going undetected. CompsAPI provides a 20-line Python script using ai=1 and confidence scores to audit filter behavior on live searches.
Read original
dev.to