A small bike has plenty of advantages - light weight, fewer pistons, especially fewer carbs, etc. But really that decision is up to you. I ride a 650 twin now (about 400 pounds wet) and had a 1968 Honda CL175 before (about 260 pounds wet) and there is definitely a difference.
And people can destroy a bike is less than 5 miles. Get a compression tester - it will tell you a LOT more than the odometer.