Your starter isn't spinning fast enough. You're experiencing the downside of CDI ignition.
Fact: You can get more initial crank speed out of a swift kick than you can ever get from a starter, even when working properly.
At its most basic, you could think of the CDI as a "charge/dump" system. It charges a capacitor based on a voltage generated at the flywheel. The faster the crank speed, the greater the initial voltage generated. It sends this voltage to a capacitor. Shortly after this occurs the crank signals this capacitor to "dump" its stored voltage and produce a spark. It then "dumps" this charged capacitor to send high voltage through the coil to the spark plug. If the initial voltage generated is never high enough to fully charge the capacitor you'll get a feeble spark or none at all when it "dumps".
It's a lot like trying to put out a fire with a 2 gallon bucket but only filling it with one cup of water at a time.
If your crank speed is too low the CDI will repeat this charge/dump/charge/dump cycle over and over without ever producing a full spark, even with the button. The TW is a wasted spark system, so it does this twice every cycle, not just on the compression stroke. Kick it, and "Voila!" More crank speed, more initial voltage, good spark.
IF you're absolutely certain your battery is good and has a full charge, all cables and grounds are tight and CLEAN at the connections both from the battery to ground and to the starter, the first place to look is at the solenoid. The TW solenoid BLOWS, the more often you've cranked on it with a low battery the sooner it will blow, and the newer the bike, the worse it blows. And those poor bastits don't have a kicker.
Fortunately the solenoid is easy to test. Get a`pair of old needlenose pliers or a jumper wire of at least 10 gauge. Turn the key "on" with the bike in neutral and jump the solenoid across the large posts with the plier tips or wire. If it starts, your solenoid is arcing and robbing juice from your starter. Replace it.
If this doesn't cure your problem you need to move on to the starter itself. Aside from a failed or weak component within the CDI itself I find most of these issues to be solenoid related, and I never replace one with a Yamaha part. I use the Kawasaki items for the KLR650, EX500 or almost any other motorcycle that has a real solenoid with a higher amperage rating on the contacts.