Main Summary

Instead of a single item encoder trained by matching loss with item text description encoder, I have implemented cross attention mechanism and contrastive learning to effectively integrate the multi-modality between item embeddings and meta data embeddings.

Sample Inference

Prompt: [User Representation] is a user representation. This user has bought "Merrell Trail Glove Barefoot Running Shoe - Men's"[HistoryEmb],"Salomon Men's XA PRO 3D Ultra 2 Trail Running Shoe"[HistoryEmb],"Hanes Men's Tagless Boxer Briefs with Comfort Flex Waistband"[HistoryEmb],"Hurley Men's Solid Phantom Boardshort"[HistoryEmb] in the previous. Recommend one next item of clothing for this user to buy next from the following item title set, "Naturalizer Women's Bola Espadrille"[CandidateEmb],"CC Junior's Rayon Camis 2 or 4 Pack"[CandidateEmb],"Champion Men's Tech Performance Boxer Brief"[CandidateEmb], …, "Marc by Marc Jacobs Women's MMJ 122/S Resin Sunglasses"[CandidateEmb],"Calvin Klein Women's Perfectly Fit Sexy Signature Demi Bra"[CandidateEmb],"OTBT Women's El Reno Bootie"[CandidateEmb]. The recommendation is

Answer: "Champion Men's Tech Performance Boxer Brief"

LLM output: "Champion Men's Tech Performance Boxer Brief"

Presentation

24_0725_연구중간보고.pdf

What I have learned

• Understanding session-based recommender system: SASREC (Preprocess)
• Creating joint collaborative item-text embeddings using autoencoder while avoiding over-smoothed representation (Original Stage 1)
• Integrating CF-based recommender model embeddings with meta data embeddings using attention mechanism and contrastive learning (Upgraded Stage 1)
• Devising an alignment network that robustly aligns item embeddings from CF-based RecSys in the token space of LLM (Stage 2)
• Transferring collaborative knowledge from CF-RecSys to the LLM without fine-tuning the LLM (Stage 2)
• Designing LLM prompt which can incorporate modality information and integrate collaborative knowledge with recommendation instructions without fine-tuning the LLM (Stage 2)
• Constructing efficient LLM framework for downstream recommendation task without fine-tuning the LLM by stably blending pretrained CF-RecSys embeddings with natural language embeddings (Overall Framework)