I don't know, in practice there are so many potential causes that you have to look case by case in situations like that. I don't have a ton of experience with the raw Claude model specifically, but would anticipate you'll have the same problem classes.
Usually it comes down to one of the following:
- ambiguity and semantics (I once had a significant behavior difference between "suggest" and "recommend", i.e. a model can suggest without recommending.)
- conflicting instructions
- data/instruction bleeding (delimiters help, but if the span is too long it can loose track of what is data and what is instructions.)
- action bias (If the task is to find code comments for example, even if you tell it not to, it will have a bias to do it as you defined the task that way.)
- exceeding attention capacity (having to pay attention to too much or having too many instructions. This is where structures output or chain of thought type approaches help. They help focus attention on each step of the process and the related rules.)
I feel like these are the ones you encounter the most.
Yes and no. I've found the order in which you give instructions matters for some models as well. With LLMs, you really need to treat them like black boxes and you cannot assume one prompt will work for all. It is honestly, in my experience, a lot of trial and error.
Usually it comes down to one of the following:
- ambiguity and semantics (I once had a significant behavior difference between "suggest" and "recommend", i.e. a model can suggest without recommending.)
- conflicting instructions
- data/instruction bleeding (delimiters help, but if the span is too long it can loose track of what is data and what is instructions.)
- action bias (If the task is to find code comments for example, even if you tell it not to, it will have a bias to do it as you defined the task that way.)
- exceeding attention capacity (having to pay attention to too much or having too many instructions. This is where structures output or chain of thought type approaches help. They help focus attention on each step of the process and the related rules.)
I feel like these are the ones you encounter the most.