Its the width of the rear tire. The TW engine uses an elongated shaft off of the engine where your sprocket sits. That little detail is a huge hinderance. Compare the TW to the XT side by side and you'll see why it would be very difficult.
As Rainman says, there are a lot of differences. The TW frame has a "bulge" on its left side to allow for the offset sprocket and the XT does not. Therefore you would need to run a "jackshaft" and two chains to clear the tire. Look up the fiche for a BW200 or BW350 and you'll see how a jackshaft is used to clear the rear tire.
Jackshafts gobble up horsepower and eat chains at a rapid rate when used on the street. To answer your question, your idea is feasible but its practicality would be limited, and would likely end up costing as much or more to build as a nice used TW.