The tribute to Whitney Houston at the 2012 Billboard Music Awards wasn't all sweet. Apparently angry members of Whitney Houston's family wanted Ray J removed from the tribute and ordered security to get him out. In case you don't remember, many members of Whitney's family, including Pay Houston, believed that Ray J was a terrible influence on Whitney.

Ray J's response? He didn't budge from his seat.

Things were about to get ugly before a security guard warned Pat Houston that things would get "physical" and get caught on tape since Ray J wouldn't voluntarily leave. Just imagine the drama if Pat Houston and Ray J got in a fight, just moments before the tribute to the woman they both love and cherish!

Later that night, Ray J said that "Pat ruined what was a great opportunity to honor the woman who I loved."

What do you think--should Ray J have been removed or not? Would love to hear your thoughts below!

If you want more, then be sure to get the full scoop on TMZ here!