In this paper a learning-based optimization method for online gear shift and velocity control is presented to reduce the fuel consumption and improve the driving comfort in a car-following process. The continuous traction force and the discrete gear shift are optimized jointly to improve both the powertrain operation and the longitudinal motion. The problem is formulated as a nonlinear mixed-integer optimization problem and solved based on adaptive dynamic programming. A major difference compared to existing approaches is that the developed control method is model-free, i.e. it does not rely on vehicle models. It can address system nonlinearities and adapt to changes in engine characteristics (e.g. consumption map) during vehicle driving. The computation is efficient and enables possible real-Time implementation. The proposed control method is studied for an urban driving cycle to evaluate the control performance with respect to the fuel economy and the driving comfort. Simulations indicate that the host vehicle can reduce the fuel consumption by 5.03% and 1.12% for two consumption maps in comparison to the preceding while keeping a desired inter-vehicle distance. The results further show a decrease of 1.59% and 2.32% in fuel consumption compared to a linear quadratic controller with the same gear shift schedule.