Leveraging LLMs for explainable recommender systems : exploring user perceptions and faithfulness in generated explanations

Abstract

Recommender systems(RSs)havebecomeubiquitous,assistingusersindiscoveringrele- vantitemsacrossvariousdomains.However,theincreasingcomplexityofRSsraisescon- cerns abouttheirtransparencyandinterpretability,particularlyinhigh-stakesapplications. This thesisinvestigatesthepotentialofLargeLanguageModels(LLMs)togenerateau- tomated, human-centeredexplanationsforRSsandassessestheirfaithfulnessinreflecting the models’internalreasoning.Weevaluatedpersonalizedmovierecommendationsand explanationsgeneratedbyGPT-3.5Turbothroughauserstudy,measuringeffectiveness, personalization, andpersuasiveness.Afollow-upstudyacrossmovie,song,andbookrec- ommendations generatedbyfourLLMs(namely,GPT-4o,Llama3,Gemma2,andMixtral 8x7B) assessedthefaithfulnessoftheseexplanationsusinganaxiomaticevaluationbased on theFeatureImportanceAgreement.OurfindingsrevealedthatwhileLLM-generatedrec- ommendations improvedusersatisfactioncomparedtorandomselections,theexplanations often failedtomeetfaithfulnesscriteria.Surprisingly,explanationsbasedonuserpreferences were notconsistentlyperceivedasmorepersonalized,effective,orpersuasivethangeneric explanations.Keycontributionsincludedauser-centricevaluationofexplanationquality, an axiomaticmethodforassessingfaithfulness,insightsintouserpreferencesandexplana- tion types,andananalysisoftheinterplaybetweenexplanationgoals.Notablechallenges identified includeLLMs’limitedpersonalizationcapabilities,variabilityinoutputsdueto non-deterministic behavior,andtheinherentblack-boxnatureofthesemodels.Thiswork highlights thepromiseandlimitationsofLLMsinExplainableRSsandprovidesafounda- tion forfutureresearchtoenhancethealignmentbetweenuserperceptionandexplanation faithfulness.