"I know Jef's throwing some shade at me a little bit, but we haven't spoken in four years," Luyendyk tells ET. "He would have lost his own bet, I guess."