Preference-aligned robot navigation in human environments is typicallyachieved through learning-based approaches, utilizing demonstrations and userfeedback for personalization. However, personal preferences are subject tochange and might even be context-dependent. Yet traditional reinforcementlearning (RL) approaches with a static reward function often fall short inadapting to these varying user preferences. This paper introduces a frameworkthat combines multi-objective reinforcement learning (MORL) withdemonstration-based learning. Our approach allows for dynamic adaptation tochanging user preferences without retraining. Through rigorous evaluations,including sim-to-real and robot-to-robot transfers, we demonstrate ourframework's capability to reflect user preferences accurately while achievinghigh navigational performance in terms of collision avoidance and goalpursuance.