In an interview with PEOPLE, the 44-year-old Sister Wives star revealed how the Brown family patriarch, 54, had been “pushing me away” since Meri, Christine and Janelle Brown left their plural ...