A leak at the exhaust manifold, while its not optimal, is not considered an air leak in the traditional sense. It will throw your test off though.
How much is the pressure dropping after five mins?
From what is sounds like, you are blocking the air at the pipe. Pull the pipe and put a plug in the exhaust manifold. You can get one at the hardware store. I use the rubber plug with a wing nut on it. Put it in the manifold and tighten it up. A stethoscope is good or a length of tubing for checking. You can also spray simple green on critical areas and look for bubbles. Check around the powervalve plugs. Those o rings are easily damaged when sliding the plugs in. Crank it up a little higher and then log the drop per minute. You are definitely on the right track.
Alright – So I got the high-temp silicone and put it b/t the manifold and cylinder and plug in the manifold again, and pressurized the engine up to 5.5psi. I let it sit for 5 mins, came back. Still at 5.5psi. Looks like its all good!