Caste is a part of privilege, but not the entire equation.

As an example, almost everyone would rather live the life of a happy poor black woman than a miserable rich white man. Regardless of race/gender, it is more a privilege to be happy than miserable. But also everyone can agree the white guy is almost always given a higher caste in society.

One of the most fucked up strategies that SJW's have had is to equate caste to privilege, because it negates any amount of personal nuance and flattens people in one of the most prejudiced ways possible (race, gender, religion, wealth).

By dissecting caste from privilege, two things are achieved: 1. Creates a place of agreement where I think most will overlap: eg the US has a caste system that's typically given more power to the rich, whites, men, and christians. 2. One can argue that any caste system which is based off race/gender is stupid; this includes Feminism.

By being against race/gender caste systems you can both agree with the inequality problem statement of SJWs (that racism/sexism are bad), and disagree with their proposed solution to just reverse the caste system (creating yet another race/gender based hierarchy.. but in the other direction!).