If you have the same ones I have then you have them backwards. The muffler slides over the header pipe. Should fit perfectly. The smaller end is the exhaust side. Also, seal the header/muffler connection with high-temp copper/silicon sealent, you can get it at any autoparts store.
That set screw holds the baffle in and should be at the rear of the bike, not near the connection.