You’re right — if travelling through just time then the above would need to be factored.

However, time is generally considered to be “space-time” which is a single fabric — and thus traversing through time will necessarily also mean moving through space.