We consider a two-stage tandem queue with two parallel servers and two queues. We assume that initially all jobs are present and that no further arrivals take place at any time. The two servers are identical and can serve both types of job. The processing times are exponentially distributed. After being served, a job of queue 1 joins queue 2, whereas a job of queue 2 leaves the system. Holding costs per job and per unit time are incurred if there are jobs holding in the system. Our goal is to find the optimal strategy that minimizes the expected total holding costs until the system is cleared. We give a complete solution for the optimal control of all possible parameters (costs and service times), especially for those parameter regions in which the optimal control depends on how many jobs are present in the two queues.