[meta] Scopes for EOD and data section(s)

[michaelblyons]

What happened?

Several languages have a concept of an "everything after this marker is literal data." To what extent should they share scopes, and what should those scopes be?

PERL

https://perldoc.perl.org/perldata#Special-Literals

while (my $line = <DATA>) { print $line; }
close DATA;
__DATA__
Hello world.

Ruby

https://docs.ruby-lang.org/en/master/Object.html#DATA

puts DATA.gets
__END__
hello world!

YAML

https://www.yaml.info/learn/document.html#end
This is not quite the same, since you can start a new document with ---.

%YAML 1.2
---
foo: bar
...
hello world

Context: #4348

[deathaxe]

In the case of YAML, punctuation.section.document.[begin|end] would probably feel more appropriate, than entity.name.document. YAML in general doesn't eagerly push contexts as those are often terminated by less indented code, which ST's syntax engine can't handle.

Perl's documentation defines __END__ as a sort of keyword, terminating actively interpreted script content. Hence it differs from __FILE__ ; constant.language not being appropriate scope for it.

Markup languages use entity.name.section, but as the focus is to denote termination of actively interpreted script, maybe generous keyword.control.flow.end might be a good candidate.

[jrappen]

Kinda sorta related, Markdown:

---
one: 1 # yaml-data-header
---

# Section heading 1

Some text.

[keith-hall]

I'm not convinced it makes sense to unify everywhere, because I think the regular expression syntax's scopes for \Q...\E make sense in that context:

Packages/Regular Expressions/RegExp (Basic).sublime-syntax

Lines 268 to 275 in 6be1771

	- match: \\Q
	scope: keyword.control.regexp
	push: literal-until-backslash-e
	literal-until-backslash-e:
	- meta_content_scope: meta.literal.regexp
	- match: \\E
	scope: keyword.control.regexp

[michaelblyons]

Keith, I can imagine multiple ways to interpret your comment. Can you elaborate whether it's one of these?

• You don't think scopes should be unified because Regex has a similar concept that doesn't make sense to unify.
• You don't think scopes should be unified, but you think that keyword.control makes sense for the __END__ keyword.
• Something else?

---

I like DeathAxe's suggestion of moving YAML to punctuation.section.document.[start|end] to match other languages with sections.

For Ruby and PERL, I think keyword.control.flow feels good in the sense that execution stops, but sort of weird in that I haven't seen anywhere with __foo__ as a keyword yet.

Conversely, entity.name.section makes sense in the "__DATA__ marks the 'data' section" interpretation. But may be missing from color schemes. 🤷

I'm open to either.

---

Any thoughts as to the literal section afterward? text.plain, meta.literal, or string.unquoted? Something else?

[keith-hall]

I meant option 1, we should consider leaving regex out the unification (unless what we come up with really fits) 🙂

keyword doesn't quite feel right when it isn't a reserved/static/constant keyword in the language grammar but user-defined based with a prefix and suffix of underscores...

entity.name.section might give it "too much status" - as in, it feels like the same section delimiter could be used multiple times (non-unique) but that scope gets in the local index, which may not be desired. Maybe it's fine?

---

For the literal section afterwards, perhaps it depends on the circumstance, whether it is verbatim or open to interpolation? For verbatim, string feels a little wrong if usual unquoted string rules don't apply, like escape characters and variable substitution.

text.plain could make sense, especially if the base scope is cleared so usual source rules (completions, comments, indentation) don't apply on that section... Conceptually, one could argue it is similar to a code fence in Markdown which doesn't specify the embedded language, so it ends up plain... (But in some cases it might be i.e. __JSON__ which is a language hint)
(I don't have any issue with meta.literal but I'm leaning towards voting for replacing the base scope unless there are drawbacks I have missed.)

[deathaxe]

Markdown frontmatter is basically just YAML + Markdown. Those --- markers are just YAML's ordinary document boundariy markers.

---

FWIW, __DATA__ and __END__ are built-in keywords in Perl and Ruby, denoting end of script. It might not be confused with HEREDOCs possibly using __END__ literal to specify boundaries. There are no arbitrary tags like __JSON__ possible in Perl/Ruby to denote special syntaxes.

The text after those keywords is reachable only by opening a special DATA file handle to read it as if it was any arbitrary text file. No interpretation nor interpolation takes place at all. Hence, clearing source main scope and replacing it by plain text would, IMHO, be most suitable. Maybe also keep main scope but just push an empty context which doesn't apply any scopes to anything.

With that regards those keywords are special to Ruby/Perl. Haven't seen any other language using keywords to terminate script content.

Hence, RegExp \K...\E is an unrelated use case in the first place. Regex in general is a unique field with regards to scope naming, I guess.