Proposal: embedded syntax block

[tinganho]

I have watched this PR #2673 and this #296 discussion for a while. But I somehow don't think it is the right approach to accept JSX directly into TS. There seem to be a demand of embedding HTML and even CSS in JS. Facebook might be the pioneer in embedding HTML in JS and therefore gotten a lot of attention recently with JSX/React. If JSX got accepted into TS, it would probably send a clear message to the JS and TS community that the TS team thinks JSX is the right way(and the only way) to go in terms of rendering views. I think it is more a correct way to accept an "embedded syntax block", so different frameworks have a chance to compete with JSX. It is a more generalized approach as oppose to accepting JSX directly into the TS.

Also remember that @ahejlsberg said "A framework is created nearly every day". So who knows if React will survive the next 2 years? Just remember how fast Angular 1 was popular and very quickly perceived as outdated(roughly a year?). That's how fast the web moves today.

So in rough terms here is my thoughts on this "embedded syntax block". I will take JSX as an example:

First, we refer to a JSX to TS transpiler. So we can transpile JSX to TS in our source code:

///<transpiler name="jsx" path="../node_modules/jsx-ts-transpiler/transpiler.ts"/>

Now, we define a JSX block we want transpiled into TS using the syntax syntax 'jsx' {...}:

let HelloMessage = React.createClass({
  render: function() {
    return syntax 'jsx' { <div>Hello {this.props.name}</div> }
  }
});

The TS's scanner stops at syntax 'jsx' { and feeds the string that reaches the closing brace } of the syntax block to the JSX-TS transpiler in this case: <div>Hello {this.props.name}</div>. The JSX-TS transpiler emits the correct TS code back along with a source map so we can map any lines and columns in the emitted TS back to our JSX. Here is how the transpiler looks like:

export transpile(source: string): Output {
  // Do something with the source
  return { output, sourceMap, diagnostics };
}

Now the TS scanner goes ahead an parse it as if it was a normal TS file.

let HelloMessage = React.createClass({
  render: function() {
    return React.createElement("div", null, "Hello", this.props.name);
  }
});

Now, we want all AST nodes outside of any "embedded syntax block" to have the same positions(pos and end) as if they where without any line and column shifts. So that we can have the correct positions on errors and warnings autocomplete etc. This is easily done by just offsetting a node's pos and end by keeping track of how much the emitted JSX-TS output shifts in character positions. For the emitted AST nodes we would marked them coming from a particular embedded syntax block(the embedded syntax block itself is an AST node with pos and end). The emitted TS nodes will be assigned pos and end beginning from 0 and not the start of its' syntax block.

Now, we have all the information we need to output the correct positions in source map and diagnostics. The diagnostics and the source map needs to be rewritten to take into consideration the new embedded syntax blocks along with its' emitted TS code.

I think this is a more open approach. If you accepts JSX directly into TS, then you are practically closing the door for any future syntax embeds and letting Facebook decide the future of syntax embeds.

---

Updates 1:

Since the TS scanner can't decide where a syntax embed block ends we would need the transpiler to help us so here is the updated transpiler:

let output = "";
let sourceMap;
let diagnostics;
let statements;

export function pushChar(ch: number): boolean {
  // Do something that produces an AST

  // Return false if it is the end of a syntax embed block else return true.
  return true;
}

export function transpile(): Output {
  // Take the AST and produce the output, source map and diagnostics.
  return { output, sourceMap, diagnostics};
}

The TS scanner pushes characters to our transpiler with pushChar() and receives true or false, whether to proceed with pushing characters to our transpiler or get the transpiled output with transpile().

The syntax for defining syntax embed could drop the syntax keyword:

'jsx' {  <div>Hello {{this.props.name}}</div> }

[Arnavion]

• How would the transformer do more complicated transformations that just string replacements on the source string? For example, how would it do a transformation that's dependent on the types of the things in the string that it's transforming, if all it has access to is the string itself?
• Even parsing the string becomes non-trivial when you consider a transform that supports embedded JS / TS (such as JSX). Then the transform function needs to invoke a full parser, perhaps invoke the TS parser in some sort of nested context.
• I guess with your proposal that type-checking, completion information, etc. will be provided by TS on the transformed code, and not by the transformer itself.
• Do braces inside the string need to be balanced? That is, can the DSL inside the syntax block have a closing brace without an open brace and not confuse the TS parser into thinking the syntax block ends earlier than it actually does?

[tinganho]

How would the transformer do more complicated transformations that just string replacements on the source string? For example, how would it do a transformation that's dependent on the types of the things in the string that it's transforming, if all it has access to is the string itself?

It's meant to transforms one string to an another. So a perfect use case is embedding HTML templates or creating your own CSS language that can live inline with your TS code. But instead of writing it in strings("..."), which React team and others like me thinks it creates "syntax pollution". Or writing it on a separate file you would need to create a separate build and tape things together which is quite a hell.

It isn't meant to be a "full" programming language to "full" programming language embed. So it isn't meant that you can embed PHP code. There is no benefit of embedding a full programming language than to write TS code alone. JSX described itself as "XML-like syntax extension to EcmaScript" — and that is what this embed functionality is meant for extending the TS language with additional syntaxes.

Even parsing the string becomes non-trivial when you consider a transform that supports embedded JS / TS (such as React). Then the transform function needs to invoke a full parser, perhaps invoke the TS parser in some sort of nested context.

The proposed embedded syntax block is only functioning as sugar. If the embed already contains TS code that are marked(in JSX it is marked inside { ... }). The one who writes the transpiler need to make sure that code isn't transformed. The transpiler doesn't need to understand TS code — just where it is. Even a trivial example let element = <MyComponent/> we can figure out what is TS code and what is JSX.

I guess with your proposal that type-checking, completion information, etc. will be provided by TS on the transformed code, and not by the transformer itself.

yes.

Do braces inside the string need to be balanced? That is, can the DSL inside the syntax block have a closing brace without an open brace and not confuse the TS parser into thinking the syntax block ends earlier than it actually does?

I missed this in my proposal. If balancing works then yes.

[Arnavion]

The transpiler doesn't need to understand TS code — just where it is.

My point is that it does. To take your example:

syntax 'jsx' { <div>Hello {this.props.name}</div> }

I assume that this will be passed as the string "<div>Hello {this.props.name}</div>" to transpile(). Then transpile() will need to interpret this string to realize that it consists of a tag "div", that then contains the string "Hello " and a TS expression "this.props.name", and then generate the appropriate codegen in your OP.

Now consider if this TS expression contains braces, double quotes, escapes, etc. Now transpile() has to implement some of the TS parser to be able to make sense of this embedded DSL. In fact, there can be more than one such {...} expression and you need to know where one ends and the next begins.

syntax 'jsx' { <div>Hello { `this.props.name${ "I'm a template string expression which has its own braces
and contains newlines. I could even contain entire JS statements inside IIFEs." } { This is an unterminated open brace` } }</div> }

transpile() will have a hard time parsing this and converting it into real TS without being able to invoke the TS parser on it.

What you're essentially describing is a very unrestricted pluggable parser-emitter, which I'm not sure is doable with a grammar such as TS. Compare this to, say, sweet.js, where the unit of transformation for the macro engine is pre-parsed tokens that vaguely resemble JS, not arbitrary text.

[tinganho]

Now consider if this TS expression contains braces, double quotes, escapes, etc. Now transpile() has to implement some of the TS parser to be able to make sense of this embedded DSL. In fact, there can be more than one such {...} expression and you need to know where one ends and the next begins.

It was a rough proposal 😄 . But one can probably make the TS evaluated only inside {{...}} or any syntax that is very distinguished from TS, in worst case even add escape characters and let the transpiler unescape them.

Instead of having balanced braces for calculating the end of the embed. All characters are feed one-by-one to a scanner/parser of the transpiler including the open brace { and ending brace } of the embed and the scanner/parser returns if is the end of the embed or not for every character.

[basarat]

///<transpiler name="jsx" path="../node_modules/jsx-ts-transpiler/transpiler.ts"/>

Perhaps better registered as transpilers using tsconfig.json.

[Arnavion]

But one can probably make the TS evaluated only inside {{...}} or any syntax that is very distinguished with TS

Yep, exactly my point. That is, your transpile() isn't a function from source string to TS. Rather, the TS parser needs to first split the text inside the syntax block into strings and expressions, and transpile() would get such a sequence of strings and expressions that it can then compose into the final TS codegen.

(And that should sound a lot like tagged template strings, because it is. Although template string tag functions deal with the results of embedded expressions, not the expressions themselves; but it's the same in that the parser needs to be aware of how to parse them and the tagger needs to work on the separate components. Of course, the value here is to retain the existing DSL rather than have to convert it to template strings.)

[tinganho]

Yep, exactly my point. That is, your transpile() isn't a function from source string to TS. Rather, the TS parser needs to first split the text inside the syntax block into strings and expressions, and transpile() would get such a sequence of strings and expressions that it can then compose into the final TS codegen.

Just for clarifications, there is no need for splits? I meant {{...}} is for TS code inside JSX and then feed every character to the JSX-TS scanner/parser and the scanner/parser returns if it is in the end of the syntax embed or not.

So an end can be calculated even if we have(no need to involve the TS parser):

syntax 'jsx' { <div>Hello {{`hello world :{  ${message`}}</div> }

The above example will generate this invalid TS code(from valid JSX code):

React.createElement('div', null, 'Hello' , `hello world :{ ${message`);

[Arnavion]

No matter how complicated you make your delimiter, it'll always be possible to use it in a way where it doesn't function as a delimiter.

syntax 'jsx' { <div>Hello {{ "hello world }} {{" }}</div> }

consists of a string "Hello" and a single expression - the string "hello world }} {{". A naive algorithm that doesn't understand TS syntax and only looks for the delimiter would instead infer two expressions.

[tinganho]

As mentioned earlier, you would need to escape }} {{, though having {{ and }} makes escaping happening rarely as oppose to just having { and }.

syntax 'jsx' { <div>Hello {{ "hello world \}\} \{\{" }}</div> }

The transpiler will then unescape those:

React.createElement('div', null, 'Hello' , "hello world }} {{");

[tinganho]

Perhaps better registered as transpilers using tsconfig.json.

True, I leave it up the TS team to decide if they want to go along this road. Then I guess, it also need to have a command line option.

[tinganho]

Side note: if possible perhaps having the shorter syntax 'transpiler' {...} by dropping the syntax keyword. So for JSX:

'jsx' {  <div>Hello {{this.props.name}}</div> }

[icetraxx]

I think a solution that didn't add any extra syntax to the language, except for the DSL that is embedded, would be much cleaner (something like Babel's transformers).