Overhaul GML highlighting

[thennothinghappened]

Changes

I've created a proper grammar for GameMaker Language, which supports many of the language's newer features, and goes a lot further than the existing one - which currently just matches a master list of keywords and function names that must be kept up to date.

The work I've done here is a port of my previous PRs to the GameMaker Manual (YoYoGames/GameMaker-Manual#160, YoYoGames/GameMaker-Manual#220), though since they use a bit of a customised fork, required some extra work to then bring upstream. As this PR encompasses the changes I listed in those PRs above I've avoided re-stating the same changes, though if it would be preferable I can bring them over into this PR's description.

Additional to the PRs however, I've also made some further minor fixes and improvements:

• JSDoc is now highlighted, in all forms stated in the GameMaker Manual.
• Multi-line #macros are now parsed properly, though this isn't currently a noticeable change.
• Many obsolete or incorrect keyword entries have been removed, and some new ones have been added.
• The gml.css theme has been updated accordingly with the changes with scopes to more correctly follow the GM IDE's colour scheme.

Comparison

As a visual overview of the changes, here's a before and after!

Before

After

Checklist

• Updated markup tests.
• Updated the changelog at CHANGES.md

[joshgoebel]

This is a huge change and will need a bit of going thru. I'll drop comments.

[joshgoebel]

What is the thinking that this type of string only scopes the string delimiters rather than the entire string?

[thennothinghappened]

I was having a lot of issues getting template literals to behave as I wanted. The issue was that if I simply set scope: "string", then the entire thing would be wrapped with that class, which would then make anything within templates, e.g.,$"{someVariable}" appear string-coloured if not claimed by another expression type (e.g., a number would still look like a number, since it would claim it.)

I might have a look at how some other languages approached this issue as I wasn't happy with this solution regardless, but I recall going down a rabbit hole of greater and greater complexity language-knowledge by the parser to fix this - there's probably an easy solution I missed though :P

[thennothinghappened]

I've come back to this again and came up with a middleground that I think is a lot more reasonable - this has been kept, but I've fixed the issue where string characters were being matched one-by-one which was making a ridiculous number of individual <span>s for each character. This has been fixed, but the string scope is still not applied to the entire span, so that the inner templates are not interfered with by string in themes.

[joshgoebel]

Maybe we'd allow some liberty with the GML theme... not sure... but typically nested items like:

string -> subst -> number

The subst color would override string, and the number color would override subst. If a theme doesn't provide a style for subst that theme is broken and should be fixed. Background can be a real problem, but not a lot of themes do crazy things with backgrounds - is GML one of them?

Still not entire I'm completely following your issue.

[thennothinghappened]

The root of the issue is that identifiers for things like local variables do not have any matching parse rule, and are unhighlighted. I didn't add any "catch all" rule at the bottom for this, though I guess I could throw in such a rule for anything that's a valid identifier with a relevance of 0. As this rule does not currently exist, here's what happens if I let the scopes be nested:

...the result being that these variables become string-coloured.
However, using the setup that's currently in, the issue is resolved:

Another thing going for this to me was that it avoids the issue where a nested scope would inherit things you don't want it to - e.g., if you had a theme which made strings bold, you don't really want the contents of their templates to also become bold with them.

Edit: I also went ahead and tried changing out the beginScope and endScope on templates to just scope. This follows suit with what JS does. While this does look acceptable and I can go this route if preferable, it still seems like a less favourable option as it diverges from the sort of highlighting you get from the GM IDE's semantic highlighting, which treats the contents of $"{in_here}" identically to were that expression written outside of a string template.

[joshgoebel]

Ok, I looked closely at your mode for string templates now - yeah this is not the preferred way - matching string like chunks inside strings. :). Neat trick though.

Isn't this same effect achievable thru pure CSS with just a ~ .string .subst rule that sets the styling (color, bg, bold/italic) back to whatever the theme considers "standard"?

This is the kind of thing I"d prefer to solve at the engine/CSS level rather than encouraging every individual grammar to start adding "hacks" like this to accomplish stylistic goals.

[thennothinghappened]

Totally understand about avoiding these sorts of hacks.

As for the alternative, if I had .subst reset the colour scheme in CSS, it'd also cause the surrounding {} to become an un-highlighted colour. I've taken that idea though and modified it such that .subst resets colours, but the {} is given char.escape instead - would this be an acceptable solution?

With a minor tweak to gml.css, this results in the intended behaviour.

[joshgoebel]

Was this wrong?

[thennothinghappened]

I removed this as the default value is already false. If it's preferable to define it explicitly, I'll put it back :)

[joshgoebel]

what's the thinking with all the removals?

[thennothinghappened]

The way the GML highlighting for function calls has worked historically has been effectively a gigantic hard-coded, and maintained by-hand list of the standard library's functions. In 2020, GameMaker 2.3 introduced a bunch of syntax changes that brought the language's syntax a lot closer to that of say, JavaScript, for instance. One of these changes was proper support of user-defined functions. This was previously done by creating assets in the IDE, and functions (or "scripts" as they were called prior to 2.3), looked a bit like this:

/// @function some_function(blah)
/// @desc Does stuff!
/// @param blah Some parameter.
/// @returns blah plus three

var blah = argument0;
return blah + 3;

Post-2.3, functions are now defined in a manner very reminiscent to JS:

/// Does stuff!
/// @param {Real} blah Some parameter.
/// @returns {Real} blah plus three
function some_function(blah) {
    return blah + 3;
}

This effectively made functions a first-class feature. As such, your average GM project nowadays uses plenty of 'em, and thus one of the very first changes I made was to highlight functions by looking for invocations: <ident>( will highlight the identifier <ident> as a function, which to my knowledge is pretty much how highlighting is done in most of the languages here. This also by extension covers function <ident>(, since there's no special handling for function other than it being a keyword.

I was, a while ago, looking into highlighting function arguments too, but that spiraled into more and more complex language parsing, i.e., due to IIFEs existing in GML just as they do in JS, and that GML supports default argument values, someone could technically have an IIFE as a default argument, thus meaning that to parse arguments would require the parser to understand a much more in-depth version of the language grammar than I am currently willing to pursue.

TL;DR: functions are first-class in modern GML, and user-functions are now commonplace, so it makes sense that this hard-coded list of function names be removed (so it doesn't need to be kept up-to-date anymore), and in its place anything looking like a function call just be highlighted as one, since there isn't any kind of semantic highlighting to decide whether that identifier is known, or anything like that here.

[tinkerer-red]

Though this does mean less maintaining, removing already existing features of the highlighter probably isn't preferred, coming from gml knowing we have arbitrary function callers like script_execute and script_execute_ext, does mean that we could still keep the list of function names and they would remain highlighted accurately.
Logic being doing nothing produces better results, so if leaving it in isnt jeopardizing anything, and it allows for catching a few extra highlights accurately then I'd say its worth keeping. Additionally getting the most up to date list is easier then ever with the GML.spec file openly available.
Though to be fair this really only means var _func = ds_grid_create; and other similar use cases wouldn't be accurately highlighted.

Given 2.3 is essentially feature complete with the coming GMRT, there shouldn't be any new functions or constants added with in some time. If I opted in to offering all of the modern function names from Monthly would we still be able to keep this if I update the list?

[thennothinghappened]

It definitely does have some value in auto-detect relevance through the GML-specific stdlib function names, but the only other cases I can think of that help would be what you pointed at above. If you're happy to take on maintaining the list I can go ahead and put it back again :)

[thennothinghappened]

@tinkerer-red I've gone ahead and re-added that built-in function list, and I've re-generated a fresh list from GmlSpec, so that's got all the new flex panel functions and whatnot.

[thennothinghappened]

Thanks for the persistence in helping me clean this up, should hopefully be up to snuff at this point!

[thennothinghappened]

I've just gone ahead and finally updated the changelog, though I noted that the next version is a patch release, so let me know if I should revert that commit until later.