Filming with Taylor Swift may have been a dream for Scott Eastwood, but for his agents, it was more of a nightmare.

Speaking to PEOPLE, the actor revealed that when he was cast as Swift's hunka-burnin'-love interest in her cinematic video for "Wildest Dreams," his agents initially advised him against working with her.

"None of my agents wanted me to do it, actually," Eastwood, who is the son of iconic Western actor Clint Eastwood, shared.

"They said, 'Oh, we don't want you to do that! Why would you be Taylor Swift's boy toy?' And I said, 'Why the hell not?'" he explained.

Though it's unclear why exactly Eastwood's agents didn't want him to work with Swift (did they believe the pop star would transform into a Cthulhu-like beast and devour him on set?), he added that the shoot was ultimately a "fantastic experience" for him, so there's no regrets.

While the video ended up performing well — it currently sits at over 480 million views since premiering in August of 2015 — the clip did face its fair share of backlash for the alleged whitewashing of the African continent.

After debuting, the Joseph Kahn-directed video was heavily criticized for "romanticizing colonialism" and for being racially insensitive by not featuring any black actors.

Hmmm, maybe Eastwood's agents had a point after all.

These Stars Were Born Wealthy: