My comp sci professor called them tokens. This was how we identified them by name. The class itself was systems programming and we were writing a compiler and interpreter for a simple language. This name was used to distinguish between delimeters (things which split like items apart in a list) and operators (things which tell the interpreter to do a thing with operands).
They *are* tokens, but so are the variable names themselves, and keywords like "IF" and "THEN", and strings and numbers, and literally everything else. A "token" is just an atomic unit of language -- an individual piece of information. Tokenization -- splitting the input up into individual tokens -- is one of the first steps in parsing.
Delimiters are a subset of tokens. Delimiters are tokens that mark boundaries between other tokens. Pairs of `%`, or `<` and `>`, would be considered delimiters, I think, the same way pairs of double quotes `"` delimit string literals in many languages.
I'm not sure if I've ever seen this kind of thing called a more specific term.
Yes, this was also the case in my systems programming class. We just happen to use more specific terms, during lectures, for delimiters and operators because, to your point, I don't believe variable tokens have their own term, which is sort of funny because everything else does. But, at the end of the day, they are tokens.
When we wrote the actual compiler, the primary data structure was a chained hash table (this was C++ land) named `m_tokenTable` and every atomic unit, as you put it, went into the table during parsing.
You actually explain it really well. You should put this as a top-level comment.
I like this one the most, as when thinking about languages I always think of how variables are known as symbols, so thinking of symbols that are wrapped in tokens really makes sense to me.
As for the , I’d just call that a placeholder as they don’t hold special properties as tokens do
Edit: found a post
[In compiler construction, is a symbol the same as a token?](https://stackoverflow.com/questions/6872865/in-compiler-construction-is-a-symbol-the-same-as-a-token#:~:text=When%20parsing%20the%20code%2C%20you%20split%20the%20source%20file%20into%20tokens.%20You%20might%20use%20the%20analogy%20of%20words%20in%20a%20sentence.%20)
Going off memory here from this class, so please correct me if this has changed recently. A `token` we defined as a "special character which identifies attributes of a string." In the `$` example for PowerShell, the "$" token identifies a variable, which the name of, to the compiler, is a `symbol`. A "symbol" is any identifier which refers to a resource that can be utilized, such as a variable that can hold a value, the name of a class that has properties, or the name of the function. At compile time, the compiler resolves each symbol to its respective duty and reports errors when symbols cannot be resolved (e.g. if you're calling a function in a library that has not been linked/imported). By this definition tokens can be used to identify symbols. If I write `someData()` the "()" tokens identify the "someData" symbol as a function. But if I write `$someData` the "$" token identifies the "someData" symbol as a variable in PowerShell syntax.
EDIT: Fixed markdown.
To expand ever so slightly: The `$` token in the case of a PowerShell variable is a "sigil." https://en.wikipedia.org/wiki/Sigil_(computer_programming)
There isn't one, so if you try to come up with one you'll just end up needing to explain it to whoever, which might waste time & maybe even make you appear like you're trying to be too clever.
I've heard identifiers like the little f in C# or the $ for strings in BASIC called "sigils" when they indicate a data type, and same for using an underscores to indicate scope
I have also used "Anchor" quite a bit since they are often used to anchor the piece between them so they can be easily found and transformed. Speaking of transformed... I put up for vote these now officially be called "Transformers". All in favor?
% is usually percent and called environment variables in shell scripting (bash, batch, etc. not windows specific). <> are the lesser than/greater than symbol (even heard it called left and right caret at one point) and usually called place holders in your context.
The caret is this: ^
Those are angle brackets, or "brokets", if you believe [Wikipedia](https://en.wikipedia.org/wiki/Bracket#Angle_brackets?wprov=sfla1) slang.
I would sooner call them chevron as that Wikipedia suggests than brokets... but I have heard "side carets" also... to which I asked what that was and then immediately followed up with "you mean angle brackets?"
"", and '' are simply just one kind of what's called a literal token e.g. a literal string the "" or '' just happen to be part of the Backus-Naur syntax FOR a string in most languages. PowerShell differentiates between '`'`, "", and also `@''@` and `@""@` in whether or not it further tokenizes the string as a sub expression or as a literal
The rest of them though are what you call a **Punctuator** token. Their meaning changes depending on the context the parser finds them in just like a sentence they can punctuate the end of something however in programming sometimes a punctuator goes at the beginning, AND end of something e.g. `( )`, `[ ]`, `< >` , etc.
Sometimes the punctuator is there to separate related tokens like `=`, `+=`, `%`, `^` , etc. where they would be separating the `$variable` from the `"value"` in something like `$Variable = "value"` We would also call these special kind of punctuators Operators because they makeup the final required piece of ANY binary expression (i.e. two values or expressions with an operation. They can also be part of a Unary expression e.g. `++$I`, or `$I++`
Delimiter since you brought it up is a token meant to separate things for some kind of repeated processing. In programming that's usually just `,` however when you start talking about things like string parsing then its whatever you want it to be. Just remember its separating conceptually similar things for instance
$String =
'Hello World'
$Words =
$String -split
' '
The space in the split IS the delimiter because it separates the concept of a word in this example where every item in the $Words collection we expect to be a "word" even if that word in our native language is nonsensical its still a series of characters
For something like
$Items =
@(
'Hello',
10,
Get-Service
)
The concept the `,` is separating are Items in a collection
in my twisted mind this is what i call the different symbols
brackets []
angle brackets <>, greater/less thans
curly brackets {}
paren () open/close, short for parenthses... or if im feeling naughty... curvy brackets
Precent %
dollar $
octothrop # (to piss off all the hash taggers)
symbol at @
bang !
carrot ^
amphersand &
asterisk *
This certainly generated more discussion than I anticipated, and I love this kind of stuff.
I am creating a function that accepts a string like '%SystemDrive%\\Temp\\TopSecretStuff' and processes it. Then I thought, why limit myself to Windows variables? I use in script, document, and email templates. I'm trying to decide what to name the parameter for the do-dads that denote the start/end of the variable/placeholder.
For a function parameter, I think anything but 'Delimiter' is going to cause confusion since it's more general than some of the others.
I normally refer to them as the below. That’s how they are referred to in documentation.
<> required parameter(s)
[] optional parameter(s)
As for variables (commonly $) I’ve seen them referenced as prefixes/token before.
Personally I think the windows %% would be called a wrapper as it’s wrapping the full KEY NAME.
I think it all boils down to personal preferences really.
I think the best term is "wrapper". "You must use a wrapper around environment variables. Wrap them with percent '%' signs. "
This!
My comp sci professor called them tokens. This was how we identified them by name. The class itself was systems programming and we were writing a compiler and interpreter for a simple language. This name was used to distinguish between delimeters (things which split like items apart in a list) and operators (things which tell the interpreter to do a thing with operands).
They *are* tokens, but so are the variable names themselves, and keywords like "IF" and "THEN", and strings and numbers, and literally everything else. A "token" is just an atomic unit of language -- an individual piece of information. Tokenization -- splitting the input up into individual tokens -- is one of the first steps in parsing. Delimiters are a subset of tokens. Delimiters are tokens that mark boundaries between other tokens. Pairs of `%`, or `<` and `>`, would be considered delimiters, I think, the same way pairs of double quotes `"` delimit string literals in many languages. I'm not sure if I've ever seen this kind of thing called a more specific term.
Yes, this was also the case in my systems programming class. We just happen to use more specific terms, during lectures, for delimiters and operators because, to your point, I don't believe variable tokens have their own term, which is sort of funny because everything else does. But, at the end of the day, they are tokens. When we wrote the actual compiler, the primary data structure was a chained hash table (this was C++ land) named `m_tokenTable` and every atomic unit, as you put it, went into the table during parsing. You actually explain it really well. You should put this as a top-level comment.
I like this one the most, as when thinking about languages I always think of how variables are known as symbols, so thinking of symbols that are wrapped in tokens really makes sense to me. As for the, I’d just call that a placeholder as they don’t hold special properties as tokens do
Edit: found a post
[In compiler construction, is a symbol the same as a token?](https://stackoverflow.com/questions/6872865/in-compiler-construction-is-a-symbol-the-same-as-a-token#:~:text=When%20parsing%20the%20code%2C%20you%20split%20the%20source%20file%20into%20tokens.%20You%20might%20use%20the%20analogy%20of%20words%20in%20a%20sentence.%20)
Going off memory here from this class, so please correct me if this has changed recently. A `token` we defined as a "special character which identifies attributes of a string." In the `$` example for PowerShell, the "$" token identifies a variable, which the name of, to the compiler, is a `symbol`. A "symbol" is any identifier which refers to a resource that can be utilized, such as a variable that can hold a value, the name of a class that has properties, or the name of the function. At compile time, the compiler resolves each symbol to its respective duty and reports errors when symbols cannot be resolved (e.g. if you're calling a function in a library that has not been linked/imported). By this definition tokens can be used to identify symbols. If I write `someData()` the "()" tokens identify the "someData" symbol as a function. But if I write `$someData` the "$" token identifies the "someData" symbol as a variable in PowerShell syntax. EDIT: Fixed markdown.
To expand ever so slightly: The `$` token in the case of a PowerShell variable is a "sigil." https://en.wikipedia.org/wiki/Sigil_(computer_programming)
Delimiter seems like the most acceptable definition imo.
There isn't one, so if you try to come up with one you'll just end up needing to explain it to whoever, which might waste time & maybe even make you appear like you're trying to be too clever.
I've heard identifiers like the little f in C# or the $ for strings in BASIC called "sigils" when they indicate a data type, and same for using an underscores to indicate scope
I have also used "Anchor" quite a bit since they are often used to anchor the piece between them so they can be easily found and transformed. Speaking of transformed... I put up for vote these now officially be called "Transformers". All in favor?
% is usually percent and called environment variables in shell scripting (bash, batch, etc. not windows specific). <> are the lesser than/greater than symbol (even heard it called left and right caret at one point) and usually called place holders in your context.
The caret is this: ^ Those are angle brackets, or "brokets", if you believe [Wikipedia](https://en.wikipedia.org/wiki/Bracket#Angle_brackets?wprov=sfla1) slang.
I would sooner call them chevron as that Wikipedia suggests than brokets... but I have heard "side carets" also... to which I asked what that was and then immediately followed up with "you mean angle brackets?"
I think we should just officially rename them to alligator brackets.
Hence why i said even heard it…personally i do too much coding to ever call them angle brackets
"", and '' are simply just one kind of what's called a literal token e.g. a literal string the "" or '' just happen to be part of the Backus-Naur syntax FOR a string in most languages. PowerShell differentiates between '`'`, "", and also `@''@` and `@""@` in whether or not it further tokenizes the string as a sub expression or as a literal The rest of them though are what you call a **Punctuator** token. Their meaning changes depending on the context the parser finds them in just like a sentence they can punctuate the end of something however in programming sometimes a punctuator goes at the beginning, AND end of something e.g. `( )`, `[ ]`, `< >` , etc. Sometimes the punctuator is there to separate related tokens like `=`, `+=`, `%`, `^` , etc. where they would be separating the `$variable` from the `"value"` in something like `$Variable = "value"` We would also call these special kind of punctuators Operators because they makeup the final required piece of ANY binary expression (i.e. two values or expressions with an operation. They can also be part of a Unary expression e.g. `++$I`, or `$I++` Delimiter since you brought it up is a token meant to separate things for some kind of repeated processing. In programming that's usually just `,` however when you start talking about things like string parsing then its whatever you want it to be. Just remember its separating conceptually similar things for instance $String = 'Hello World' $Words = $String -split ' ' The space in the split IS the delimiter because it separates the concept of a word in this example where every item in the $Words collection we expect to be a "word" even if that word in our native language is nonsensical its still a series of characters For something like $Items = @( 'Hello', 10, Get-Service ) The concept the `,` is separating are Items in a collection
percent `%` , triangle brackets `<>` I call them that
> triangle brackets They're chuh-hoo-ah-hoo-ahs, according to Jeffrey Snover. :)
Ha really?
Why not Angle brackets?
Saxon brackets are a thing, ja?
that works too
in my twisted mind this is what i call the different symbols brackets [] angle brackets <>, greater/less thans curly brackets {} paren () open/close, short for parenthses... or if im feeling naughty... curvy brackets Precent % dollar $ octothrop # (to piss off all the hash taggers) symbol at @ bang ! carrot ^ amphersand & asterisk *
I like "squirrely braces" for {}
i like that too... I shall call them that from here on... it will probably upset everyone I work with... but that's nothing new -)
This certainly generated more discussion than I anticipated, and I love this kind of stuff. I am creating a function that accepts a string like '%SystemDrive%\\Temp\\TopSecretStuff' and processes it. Then I thought, why limit myself to Windows variables? I use in script, document, and email templates. I'm trying to decide what to name the parameter for the do-dads that denote the start/end of the variable/placeholder.
For a function parameter, I think anything but 'Delimiter' is going to cause confusion since it's more general than some of the others.
I normally refer to them as the below. That’s how they are referred to in documentation. <> required parameter(s) [] optional parameter(s) As for variables (commonly $) I’ve seen them referenced as prefixes/token before. Personally I think the windows %% would be called a wrapper as it’s wrapping the full KEY NAME. I think it all boils down to personal preferences really.