This is one of those things Apple says is perfectly fine but still seems a little creepy. Your iPhone keeps track of where you've been and stores that information on a map buried deep in your phone's settings.
To find this map, open your phone's settings and choose privacy, then select location services.
At the bottom of the list is 'system services', and toward the bottom of that screen is a tab marked "frequent locations."
If it is turned on, and it probably is unless you've gone in and turned it off, all of your most frequented addresses and places are plotted out on a map, categorized by city.
The entries seem to be limited to the past 3 months but lists the days you were there, the time of day you arrived and when you left.
Apple says the information is used to learn places that are significant to you in order to provide personalized services such as predictive traffic routing. Apple also states that the data is kept solely on your device and isn't sent to Apple without your permission.
If it still creeps you out a bit, you can turn it off by toggling the frequent locations tab. From what we were able to gather, you won't be inconvenienced by turning it off and won't lose any functionality with maps and direction.
You can also keep frequent locations turned on and clear the map of your frequent locations from time to time.
Apple isn't the only company keeping a record of your frequently visited places, Google keeps an eye on where you've been with its own map whether you're using an iPhone or Android device.