The game takes place in Castle Oblivion, a mysterious castle kept by Marluxia. Sora and his teammates are told that the castle causes visitors to lose their memories upon entering. The lobby and areas between floors are white with flower-themed decorations, but each floor can be transformed into a different world from the first Kingdom Hearts game using "world cards" created from Sora's memories. Like before, many of the worlds are based on Disney films while the rest were created by Square Enix.
Unlike the previous game, the worlds are created from Sora's memories. As such, many of the events experienced in Kingdom Hearts are relived in this game; Sora encounters memory-based versions of Disney characters that he has met before. The individual plotlines differ from those in the original game and revolve around the theme of memory. The exception is the Deep Jungle world which does not reappear in Chain of Memories. Just as Kingdom Hearts had several worlds created specifically for that game, Chain of Memories introduces Twilight Town, in addition to the original worlds of Kingdom Hearts. The world card for Twilight Town was created from memories on "the other side of Sora's heart".