FWIW, the main goal of Light Table is to provide a platform for people to experiment with and build these kinds of experiences without having to worry about all the other standard editory stuff. By using a browser for the UI, we have the opportunity to very easily play with all sorts of really interesting display and editing paradigms.
Unfortunately, the notion of editing an AST is by definition language specific, so it's unlikely that someone could create the structural editor and just have it work for everybody. Moreover, as they mention here for Lamdu, it often requires looking at the language slightly differently and enforcing some rules that wouldn't normally exist in the code. But, with a decent foundation we can at least make writing the necessarily language specific parts relatively straightforward.
For those new to this world and wondering why this stuff doesn't seem to exist, one of the biggest problems with projectional editors is handling the translation problem. How do you reconcile changes in your representation with changes in the underlying code? Can you reliably parse handwritten code into your AST representation write it back and so on without any loss? What happens with handwritten styles that maybe don't fit into the projection's way of viewing the world? What if I know a better way to output the AST than you? It's also particularly difficult dealing with change over time since there are no true unique identifiers for bits of code.
The approach they're taking is likely the correct approach for the "future" - we should be designing languages and approaches to coincide with the ability to tool them. Even better would be to never have a handwritten format at all: the cannonical representation is always the AST. And though you might be able to edit a projection of it as text, you're never at a loss for how to get back to your "good" representation. This is the world I think we ultimately need to get to and you'll be seeing some really cool stuff from us in that vein early in 2014.
The main problem with structured editing is that it requires programmers to write code in a certain order that is often goes against their "flow;" enforcing a rigid flow is a huge usability negative. Even without structured editing, we see cases where increased flexibility improves flow; e.g. in dynamically typed languages. Unfortunately, improvements in flow often lead to reductions in feedback (e.g. via static typing).
There are things we can do to have our cake (flow) and eat it (feedback) also. Code completion was one of the great boons of structured editing (introduced in Alice Pascal circa 1985), and by the late 90s we learned how to create "language aware" editors that could leverage this feature without flow disrupting structured editing. The same goes for static typing: we can, through some heavy type inference, infer semantic information responsively while the user is typing, and use that to provide responsive feedback.
I'm in the camp where the programming experience should be considered holistically. The IDE is a part of that experience, and so language design should occur concurrent with IDE design. With some smart incremental compilation magic along with language-specific rendering in the IDE, we can build programming experiences that provide the benefits of structured (and projectional) editing without the flow costs. Or at least, that is the premise of my research :)
There's a structured editing mode for Emacs called Paredit, to use for s-expression languages like Lisp, Racket, Clojure.
Since the AST for such languages is s-expressions, and some people struggle with parens, this is an interesting "fun size" example of editing the AST.
The thing is, paredit is quite challenging for most people to adopt, due to what you mentioned about the "flow" they've already learned. Magnar Sveen has a great video about this at http://www.youtube.com/watch?v=D6h5dFyyUX0
Best quote: "If you think paredit is not for you, then you need to become the kind of person that paredit is for."
EDIT: To clarify, Magnar is quoting technomancy a.k.a. Phil Hagelberg.
Yes, Paredit's not really a straitjacket, but your editor having new concepts — verbs which operate on code units.
It overrides some deletion commands to maintain your code well-formed, but there's simple ways (like cutting text) to break those rules, so it's ill-formed. (Currently, paredit won't start if it sees you're editing ill-formed text. But if it's already started, it'll continue running.)
Right, I realize it's not an inescapable straitjacket, and I'm definitely not putting down paredit; upvoted your comment.
I mention it because it's an example of structured editing of an extremely simple AST. And even people who want to use it (who want to leverage the so-called "straitjacket"), often find it quite hard to change their flow and adapt it.
That seems not to bode well for this being enthusiastically used generally. OTOH I suppose you could argue that a lot of people like auto-completing IDEs, and/or hate to type, so who knows.
The main problem with structured editing is that it requires programmers to write code in a certain order that is often goes against their "flow;" enforcing a rigid flow is a huge usability negative.
I agree that structured and visual editing paradigms have failed. But ultimately all you're saying here is that structure approaches haven't offered enough positive to offset the cost of learning them. That may be true but it would be good to know why.
My guess is that the data structures generated by the text of programs is just too varied and complex for any visual representation system to be of use. The programmer is interested in the final structure of their application and every intermediate one. Creating a program to represent those visually should be harder than implementing the application. Multiply that by every application that could exist.
> But ultimately all you're saying here is that structure approaches haven't offered enough positive to offset the cost of learning them. That may be true but it would be good to know why.
No, I'm claiming that bad flow, even after you've learned and adapted to a structured paradigm, needs to be solved directly. Even if you can provide really great feedback, the bad flow will still exist and cause inefficiencies not offset by enhanced feedback, which is aimed at an orthogonal set of problems.
Edit: a good analogy might be between agility and armor in a tank. You can obviously make the tank more survivable by improving either one, but its overall usage becomes more limited when emphasizing one feature at the expense of the other.
"Unfortunately, the notion of editing an AST is by definition language specific, so it's unlikely that someone could create the structural editor and just have it work for everybody.
"
This stuff has been done forever, and it's been "the next generation" forever. Things like separation of structural editing from the language specficness of an AST while still providing a sane and useful editor are paths that have been tread before.
Not that this means people shouldn't try, but as you can see from just these two links, there has been a lot of research and effort put into this area in general. This is an area where i'd read a lot more than i'd sit down and code, because a lot of smart people have been this way before.
Case and point: Susan Graham (and maybe Tim Wagner), who lead the Harmonia project (and many projects before that), is basically the goto person in the world if you ever wanted to know anything about incremental analysis of programming languages (she also helped write gprof, among other things)
Harmonia was more about language-aware editing, Ensemble and Pan were more about structured editing.
> This stuff has been done forever, and it's been "the next generation" forever.
The Harmonia stuff never worked that well when I tried it. I've also read many of Graham and Wagner's papers on their tooling, and you know what....
> This is an area where i'd read a lot more than i'd sit down and code, because a lot of smart people have been this way before.
...it turns out we can do much better than what they did given per language consideration. Once I decided not to use Harmonia for the Scala IDE, I came up with a way to incrementalize Martin's compiler fairly easily. It wasn't that hard in hindsight, but none of those fancy algorithms were really necessary at all! Instead, we just needed an incremental computation framework like Glitch [1] to do the heavy lifting transparently, which eventually led to my work in live programming.
So the moral of this story: don't trust everything you read from an academic conference. Often work is incomplete, not appropriate for your context, outdated, or even unworkable. Usually, its incomplete and there is lots of room for improvement.
I don't think it can be called true or false that easily. Sure you can translate multiple languages into one AST representation (maybe embellished to allow round-tripping). But is it still the same language? I would say no. But I would go further and say that not even a language specific AST is the same language.
I think the idea that we can seperate programming languages from their surface syntax and still claim they are the same languages is misguided. Yes programming languages have semantics, but the user interface aspect is an inherent part of their identity because it is an inherent part of what it means for humans to use language. There is no such thing as syntactic sugar.
There's been some interesting work by Conor McBride on Epigram where he uses structural editing to help communicate the proof burdens that the compiler can infer and what remains for the user to do. Epigram is long gone, but there are still papers describing how it worked.
This is what a next-generation programming environment must have. Primarily, one manipulates AST nodes, projected visually into lines of code. But, to preserve the comfort and flexibility of free-form text editing, the editor knows about non-AST text fragments, tree "holes", and other representations of intermediary editing state. (A big UI problem for sure!)
An editor that can deal with the ways in which code changes over time elegantly is what we really need. Since editing a plaintext projection discards code movement metadata, editing the AST directly is paramount. Without that metadata, the programming environment can't version control structured code automatically.
An approach that might work with existing languages is:
To use a traditional flat editor (not a structured editor), but show inferred information around the code. I am thinking of visuals that are akin to http://explainshell.com/ but a lot more compact.
The inferred information need not be shown always; it could be shown dynamically based on the user's context (current line, current function, etc)
>> Can you reliably parse handwritten code into your AST representation write it back and so on without any loss? What happens with handwritten styles that maybe don't fit into the projection's way of viewing the world?
Genuinely asking, is this really a problem? It sounds solvable to me.
The approach they're taking is likely the correct approach for the "future" - we should be designing languages and approaches to coincide with the ability to tool them.
Unfortunately, we still have many in the PL community that don't think that way.
key quote: "In fact, the most powerful languages may initially have the least powerful tool support. The reason for this is that the language developer, like the language adopter, has to make a choice: whether to dedicate limited development resources towards language features, or towards tool support."
Unfortunately, the notion of editing an AST is by definition language specific, so it's unlikely that someone could create the structural editor and just have it work for everybody. Moreover, as they mention here for Lamdu, it often requires looking at the language slightly differently and enforcing some rules that wouldn't normally exist in the code. But, with a decent foundation we can at least make writing the necessarily language specific parts relatively straightforward.
For those new to this world and wondering why this stuff doesn't seem to exist, one of the biggest problems with projectional editors is handling the translation problem. How do you reconcile changes in your representation with changes in the underlying code? Can you reliably parse handwritten code into your AST representation write it back and so on without any loss? What happens with handwritten styles that maybe don't fit into the projection's way of viewing the world? What if I know a better way to output the AST than you? It's also particularly difficult dealing with change over time since there are no true unique identifiers for bits of code.
The approach they're taking is likely the correct approach for the "future" - we should be designing languages and approaches to coincide with the ability to tool them. Even better would be to never have a handwritten format at all: the cannonical representation is always the AST. And though you might be able to edit a projection of it as text, you're never at a loss for how to get back to your "good" representation. This is the world I think we ultimately need to get to and you'll be seeing some really cool stuff from us in that vein early in 2014.