There is nearly zero chance that any sort of frame damage is causing your problem. 3-4 mm would be a lot of side play, so first be sure the assembly is correct and you understand how it is supposed to work. Your picture looks like the swing arm bolt is not tight. That end cap should be tight against the frame and not be able to rotate with the swing arm. You should be able to remove that much gap just by tightening the swing arm bolt. Think about the assembly without the swing arm (and the pressed in bushings) installed. There would be just the frame, the big swing arm bolt, the center spacer, and the two end caps/grease seals. If you assemble them without the swing arm, you will see that the spacer and end caps get bolted tightly between the sides of the frame and do not rotate. There can be some variance in frames, some being a super tight fit to get these parts between the two frame sides, and some not so tight. Regardless, the concept is to clamp this assembly tightly between the frame sides making it sort of an additional structural frame element. If the frame is tight, it can be very difficult to place all the parts inside it without scratching up the paint, and if loose, it may require considerable tightening of the swing arm bolt to draw the frame sides up tight. On some old bikes I have machined thin spacers to take up the play, but usually just tighten it up and all is well. The swing arm and bushings float on this rigid assembly, hopefully with no discernible play. To get that, you usually need new bushings which press into the swing arm, and sometimes a new spacer. Some bikes have separate internal bushings and a spacer between them, but your bike has a single part combining the three. It is a fixed precise length, usually hardened steel where the bushings ride, and unlikely to be worn terribly unless rust is involved. Since it is the internal bearing surface, it needs to be replaced if worn. So, assembled with new bushings and spacer/inside bushings, your swing arm should have zero "up and down" play, and if super lucky, no side to side play. It is possible for the swing arm and bushings to be too wide and get clamped down before the swing arm bolt is tightened down against the end caps and center spacer. This can happen if the bushings are not completely pressed all the way into the swing arm, or if the bushings are aftermarket and simply not dimensioned correctly. If this occurs, material must be machined from the ends of the bushings to gain clearance. If the swing arm can still be slid back and forth between the frame sides (actually between the two end caps) you will need to add shims under the end caps (between the bushing and end cap). This is typically a very small adjustment and a trial and error process. We are talking about paper thin shims here, but may have to stack up to a mm or two in severe cases. Shims are commonly hard to get, but the factory used them and they can usually be had from a dealer with experienced parts staff. Often they are not shown in exploded views, so you need someone that knows what they are doing. I usually set up with minimal grease, just enough to glue the parts together while assembling, and when I figure out the right amount of shims, Grease everything well. When correctly assembled, your swing arm should have zero play in any direction, and be held tightly enough in the frame to barely fall from its own weight when unsupported, bare without the wheel. Once done you can add grease with a grease gun. It is definitely a PITA to get right, but really pays in the end. Most old bikes drive and handle very surprisingly well when tight and fresh, and the swing arm pivot make all the difference!
