tl;dr: Misalignment happens if two intelligent entities prefer mutually exclusive outcomes. Because the world can ever only be in one state at any given time, misalignment must be resolved to determine the outcome. If the power distribution is very skewed, the outcome will usually be skewed towards the preferences of the more powerful, as well.
Epistemic state: Very obvious to me.
Imagine you are in a room with a friend. After a while, you find that the room urgently needs airing and open the window. Unfortunately, you live in a country where it's freezing cold outside, which makes your friend complain about how cold it is and immediately go to the window to close it again.
Everyone who has siblings knows what I'm talking about. You want A, your sibling wants not A. Then a fight starts, which determines whether A or not A. Such a disagreement about the outcomes is called misalignment. If both parties want the same outcome, we call them aligned to each other in this particular decision. In other words, alignment is the degree to which they agree on how the world should look like. Accordingly, misalignment happens if two intelligent entities want mutually exclusive outcomes.
These entities are not even required to be particularly intelligent. They only must have preferences, such that they can disagree on the desired outcome. They are not even required to be aware that the other party exists that disagrees with them. Take for example viruses. Viruses are pretty dumb, to our standards, and they are not "aware" of our preferences in any meaningful way. Nevertheless, they have the clear preference of more of them being in the world, while you, or better your immune system, has the opposite preference.
Misalignment often leads to an alignment problem, because the world can be only in one state at any given time. To solve an alignment problem means to resolve a misalignment to determine the outcome. This is achieved via negotiation, in the widest sense. By negotiation I don't only mean a compromise, an exchange, or maybe a fierce discussion, in the classical sense. It could, more broadly, also be following an implicit social rules, providing incentives, or other strategies. If power is unequally distributed, the result of the negotiation usually tends to be more skewed towards the preference of the more powerful. In more extreme cases, the more powerful can decide on the outcome single-handedly, and the negotiation consists more or less of how much resistance the other party dares to exert.
That's also why power corrupts. Power allows you to make things more like you want them, and less like others want them. People often mistake their actual behavior for what they really want. This is because their behavior is a result of negotiation and humans are really good at convincing themselves that they actually wanted what they did. Power doesn't actually change your deepest preferences, it just makes them more visible. If you want to see who people really are, give them power. One effective measure for humans to fight power corruption is to activate the built-in mechanism to equalize perceived power: empathy.
Can you find occasions in your life where you had to cut down on your needs, because you were overpowered by someone else? Or where you have been able to turn a table to your advantage? Do you think the use of empathy could have led to better solutions? Or, more generally, can you think of other situations where misalignment happens? Here are some examples:
- In the window example in the beginning of this post there is a misalignment between you and your friend
- You might not be even clear within yourself whether you prefer warm or fresh air
- Oil companies want to maximize profit, but humans don't want the consequences due to climate change
- A particular artificial intelligence wants to maximize ad-revenue, but humans don't want to be subtly manipulated
- Some animals held captive in factory farms might want to leave their cages, but humans don't want them to leave
- An unfortunate incentive structure might lead to absurd outcomes
- Feminism is to a large part about redistributing power from men to non-men to make the outcomes include a fair consideration of non-men's preferences
- More generally, groups according to some identity attribute have some preferences that are different from the rest of the population
- A government in power implements laws that aren't in the interest of the population
- One country wants to extend its territory, while the other doesn't want to lose it
Edit: I changed the last paragraph to connect a bit better to the text.