Rivera was born at Beth Israel Medical Center in New York City, New York, the son of Lillian (née Friedman), a waitress, and Cruz "Allen" Rivera (October 1, 1915 – November 1987), a restaurant worker and cab driver. Rivera's father was a Catholic Puerto Rican, and his mother is of Ashkenazi Russian Jewish descent. He was raised "mostly Jewish" and had a Bar Mitzvah. He grew up in Brooklyn and West Babylon, New York where he attended West Babylon High School. He has four siblings: Wilfredo, Sharon, Irene, and Craig.