Although I agree with the notion that deep knowledge of complex systems has decreased over the years, I'd argue that the problem isn't the lack of interest in gaining deeper knowledge that's the problem, nor a perceived lack of respect for expertise.
I'd argue its more to do with the nature of how large codebases morph over time as they're retrofitted to do more usercases. Its a lot easy in the short term to just hack in the functionality you need into an existing codebase, instead of rewriting the entire thing from scratch every time. This comes at the cost of maintainability and leads to a lot of headaches in the long term, as systems that where never designed to work together clash in unexpected ways.
I don't begrudge anyone for not wanted to dive into codebases like that.
> I feel like "Complexity" is the wrong word here. [...] I'd argue its more to do with the nature of how large codebases morph over time as they're retrofitted to do more usercases. Its a lot easy in the short term to just hack in the functionality you need into an existing codebase, instead of rewriting the entire thing from scratch every time. This comes at the cost of maintainability and leads to a lot of headaches in the long term, as systems that where never designed to work together clash in unexpected ways.
Rich Hickey separates complexity into two parts[0], accidental and intentional (IIRC). Intentional complexity is complexity that stems from the domain of your problem, there isn't much you can do to manage it, DNA splicing will be complex no matter what you do, for example.
The other part is accidental complexity, which is complexity you could deal with, but for whatever reason choose not to right now, as you touch on in your comment. This eventually snowballs into a huge spaghetti monster as you pile hacks on top of hacks.
There is a (famous at this point) talk from Hickey where he dives much deeper into how he sees complexity, which happens to be a really good pitch for Clojure, but I first saw it before I knew Clojure, and the perspective applies to software in general, so no Clojure knowledge needed.
Although I agree with the notion that deep knowledge of complex systems has decreased over the years, I'd argue that the problem isn't the lack of interest in gaining deeper knowledge that's the problem, nor a perceived lack of respect for expertise.
I'd argue its more to do with the nature of how large codebases morph over time as they're retrofitted to do more usercases. Its a lot easy in the short term to just hack in the functionality you need into an existing codebase, instead of rewriting the entire thing from scratch every time. This comes at the cost of maintainability and leads to a lot of headaches in the long term, as systems that where never designed to work together clash in unexpected ways.
I don't begrudge anyone for not wanted to dive into codebases like that.