Yes, the Elders did take the memories of the community. The Elder's believed it would be morally right to exact each individual into what they called, Sameness. They also took colour, variety, and freewill away from the citizens of The Community.
Yes the Elders took all of the towns memories. They also bestowed them all to the one man, the "Giver". I hope my answer has helped you. (Also might wanna brush up on grammar lessons)
