Data-driven Koopman operator approximation has gained interest recently for its ability to embed nonlinear systems into a lifted linear state space using only measurements. When control inputs are included, however, the lifted dynamics render a bilinear form, which poses challenges for controller synthesis, such as Model Predictive Control (MPC). This paper proposes an on-policy actor-critic Deep Reinforcement Learning (DRL) framework that simultaneously learns the Koopman bilinear dynamics and an MPC neural cost map. Instead of directly generating control actions, the actor network takes the Koopman-lifted states and produces MPC weight matrices for each prediction step. These state-dependent weight matrices serve as high-level guidance for the control objective, allowing the low-level MPC to run under very short prediction horizon while maintaining stability and enforcing safety constraints. Simulations carried out with the OpenAI Gym library demonstrate that, without requiring explicit knowledge of the dynamics, the proposed Actor-Critic Koopman MPC (ACKMPC) achieves control accuracy and disturbance robustness on par with a model-based ACMPC, and outperforms a pure DRL-learned policy using baseline Proximal Policy Optimization (PPO). It also exceeds standard Koopman MPC (KMPC) in both robustness and computational efficiency.