Explainability towards users is essential if there is any sort of responsibility for what the model produces.

But explainability at a deeper level is also critical towards developers, to close the feedback loop and iterate towards better models.