STscript Parser Rewrite (#1965)

* set isForced to true on input

* make floating auto-complete follow horizontal scrolling

* add callable closure vars

* changes to /let and /var for callable closures

* fix error message

* fix scope for closure arguments

* if should return the pipe result from closures

* use /run to call closures and no arguments on immediate closures

* throw exception from QRs window-function if no match

* when to show autocomplete vs info only

* autocomplete positioning

* autocomplete styling

* add theming to autocomplete (theme, dark, light)

* improve autocomplete show/hide logic and editor selection

* use blur tint color instead of chat tint color and use blur setting

* cleanup and docs

* use scope macros for QR args

* add enter to select autocomplete

* fix no executor found

* cleanup and comment

* fix alias list in help string

* fallback to empty string piped value if null or undefined

* fix typo

* blur textarea on ctrl+enter execute (and refocus after)

* stop executeSlashCommand if parser throws

* move /let and /var callbacks into functions

* switch textarea to monospace when value starts with slash

* add double pipe a pipe breaker

* fix /? slash

* remove some logging

* add "/:name" as shorthand for "/run name" after all

* move shit around

* fix error message

* use testRunShorthandEnd

* use parseQuotedValue and parseValue to determine name for "/:"

QR labels and set names can include spaces

* add some adjustments to make autocomplete work properly

some hint in there about "/:" would still be nice

* add autocomplete style  selector

* only strip quotes from subcommand if they are at both ends

* fix JSDoc

* escaping

* allow open quotes on dry run

* throwing shit at the wall for /: autocomplete

* escapes only for symbols

* clean up autocomplete

* improve performance

* fix scope macros

* remove unescaping of pipes

* fix macros in scope copy

* fix "/? slash"

* don't run parser for getNameAt if text has not changed

* fix options filter

* re-enable blur listener

* restore selection on non-replace select

* fix for escaping first character of value

* add support for {{pipe}} and {{var::}} closures

* add index support to var macro

* add scoped var macro to macro help

* more escape fixes

* reduce autocomplete render debounce

* cleanup

* restore old escape handling and parser flag for strict escaping

* fix "no match" autocomplete message

* add dummy commands for comments and parser flag

* fix type annotations

* somewhat safer macro replacements

* fix autocomplete select on blank / "no match"

* fix cutting off handled part in substitution

* add parser flag REPLACE_GETVAR

Replaces all {{getvar::}} and {{getglobalvar::}} macros with {{var::}}.
Inserts a series of command executors before the command with the macros that:
- save {{pipe}} to a var
- call /getvar or /getglobalvar to get the variable used in the macro
- call /let to save the retrieved variable
- return the saved {{pipe}} value

This helps to avoid double-substitutions when the var values contain text that could be interpreted as macros.

* remove old parser

* fix send on enter when no match

* deal with pipes in quoted values (loose escaping)

* add default parser flags to user settings

* allow quoted values in unnamed argument

* set parser flag without explicit state to "on"

* add click hint on parser error toast

* dirty more detailed cmd defs

* remove name from unnamed arg

* move autocomplete into class and floating with details

* replace jQuery's trigger('input') on #send_textarea with native events because jQuery does not dispatch the native event

* fix ctrl+space

* fix arrow navigation

* add comments

* fix pointer block

* add static fromProps

* fix up dummy commands

* migrate all commands to addCommandObject

* remove commented comment command

* fix alias in details

* add range as argument type

* switch to addCommandObject

* switch to addCommandObject

* fix height

* fix floating details position on left

* re-enable blur event

* use auto width for full details on floating autocomplete

* auto-size floating full details

* fix typo

* re-enable blur listener

* don't prevent enter when selected item is fully typed out

* add autocomplete details tooltips

* add language to slash command examples

* move makeItem into option and command and fix click select

* use autocomplete parts in /? slash

* fix alias formatting

* add language to slash command examples

* fix details position on initial input history

* small screen styles

* replace registerSlashCommand with detailed declarations

* put name on first line

* add missing returns

* fix missing comma

* fix alias display in autocomplete list

* remove args from help string

* move parser settings to its own section

* jsdoc

* hljs stscript lang

* add hljs to autocomplete help examples

* add missing import

* apply autocomplete colors to stscript codeblocks (hljs)

* add fromProps

* cache autocomplete elements

* towards generic autocomplete

* remove unused imports

* fix blanks

* add return types

* re-enable blur

* fix blank check

* Caption messages by id

* add aborting command execution

* fix return type

* fix chat input font reset

* add slash command progress indicator

* add missing return

* mark registerSlashCommand deprecated

* why??

* separate abort logic for commands

* remove parsing of quoted values from unnamed arg

* add adjustable autocomplete width

* revert stop button pulse

* add progress and pause/abort to QR editor

* add resize event on autocomplete width change

* add key= argument to all get vars

* refactoring

* introduce NamedArgumentAsignment

* add TODOs

* refactoring

* record start and end of named arg assignment

* refactoring

* prevent duplicate calls to show

* refactoring

* remove macro ac

* add secondary autocomplete and enum descriptions

* add syntax highlighting to QR editor

* add enum descriptions to /while

* add /let key=... to scope variable names

* add unnamed argument assignment class and unnamed argument splitting

* fix QR editor style

* remove dash before autocomplete help text

* add autocomplete for unnamed enums

* fix remaining dom after holding backslash

* fix for unnamed enums

* fix autocomplete for /parser-flag

* add parser-flag enum help

* fix type annotations

* fix autocomplete result for /:

* add colored autocomplete type icons

* collapse second line autocomplete help if empty

* mark optional named args in autocomplete

* fix when what

* remove duplicate debug buttons

* dispatch input on autocomplete select

* prevent grow from editor syntax layer

* add auto-adjust qr editor caret color

* remove text-shadow from autocomplete

* join value strings in /let and /var

* add /abort syntax highlight

* fix attempting secondary result when there is none

* rename settings headers and split autocomplete / stscript

* add parser flag tooltips

* add tooltips to chat width stops

* fix typo

* return clone of help item

* fix enum string

* don't make optional notice for autocomplete arguments smaller

* avoid scrollbar in chat input

* add rudimentary macro autocomplete

* strip macro from helptext

* finally remove closure delimiters around root

* cleanup

* fix index stuff for removed closure delimiters

* fix type hint

* add child commands to progress indicator

* include sub-separator in macro autocomplete

* remove all mentions of interruptsGeneration and purge

* remove unused imports

* fix syntax highlight with newline at end of input

* cleanup select pointer events

* coalesce onProgress call

* add regex to STscript syntax highlighting

* fix closure end

* fix autocomplete type icon alignment

* adjustments for small screens

* fix removing wrong element

* add missing "at=" arg to /sys, /comment, /sendas

* add font scale setting for autocomplete

* add target=_blank for parser flag links

* fix for searching enums

* remove REGEXP_MODE from hljs
just causes trouble

* fix autocomplete in closures

* fix typo

* fix type hint

* Get rid of scroll bar on load

* Add type hint for /send name argument. Fix 'at' types

* Add 'negative' arg hint to /sd command

* reenable blur event

* Allow /summarize to process any text

* Compact layout of script toggles

* Expand CSS by default

* fix double ranger indicator and adjust to narrow container

* make custom css input fill available vertical space

* reduce scroll lag

* use default cursor on scrollbar

* Clean-up module loading in index.html

* fix tab indent with hljs

---------

Co-authored-by: Cohee <18619528+Cohee1207@users.noreply.github.com>

public/script.js

@@ -158,7 +158,7 @@ import {
 import { debounce_timeout } from './scripts/constants.js';
 
 import { ModuleWorkerWrapper, doDailyExtensionUpdatesCheck, extension_settings, getContext, loadExtensionSettings, renderExtensionTemplate, renderExtensionTemplateAsync, runGenerationInterceptors, saveMetadataDebounced, writeExtensionField } from './scripts/extensions.js';
-import { COMMENT_NAME_DEFAULT, executeSlashCommands, getSlashCommandsHelp, processChatSlashCommands, registerSlashCommand } from './scripts/slash-commands.js';
+import { COMMENT_NAME_DEFAULT, executeSlashCommands, executeSlashCommandsOnChatInput, getSlashCommandsHelp, isExecutingCommandsFromChatInput, pauseScriptExecution, processChatSlashCommands, registerSlashCommand, stopScriptExecution } from './scripts/slash-commands.js';
 import {
     tag_map,
     tags,
@@ -227,6 +227,10 @@ import { currentUser, setUserControls } from './scripts/user.js';
 import { callGenericPopup } from './scripts/popup.js';
 import { renderTemplate, renderTemplateAsync } from './scripts/templates.js';
 import { ScraperManager } from './scripts/scrapers.js';
+import { SlashCommandParser } from './scripts/slash-commands/SlashCommandParser.js';
+import { SlashCommand } from './scripts/slash-commands/SlashCommand.js';
+import { ARGUMENT_TYPE, SlashCommandArgument } from './scripts/slash-commands/SlashCommandArgument.js';
+import { SlashCommandBrowser } from './scripts/slash-commands/SlashCommandBrowser.js';
 
 //exporting functions and vars for mods
 export {
@@ -1716,6 +1720,7 @@ export async function reloadCurrentChat() {
  */
 export function sendTextareaMessage() {
     if (is_send_press) return;
+    if (isExecutingCommandsFromChatInput) return;
 
     let generateType;
     // "Continue on send" is activated when the user hits "send" (or presses enter) on an empty chat box, and the last
@@ -2397,30 +2402,14 @@ export async function generateQuietPrompt(quiet_prompt, quietToLoud, skipWIAN, q
  * @param {string} message Text to be sent
  * @returns {Promise<boolean>} Whether the message sending was interrupted
  */
-async function processCommands(message) {
+export async function processCommands(message) {
     if (!message || !message.trim().startsWith('/')) {
         return false;
     }
-
-    const previousText = String($('#send_textarea').val());
-    const result = await executeSlashCommands(message);
-
-    if (!result || typeof result !== 'object') {
-        return false;
-    }
-
-    const currentText = String($('#send_textarea').val());
-
-    if (previousText === currentText) {
-        $('#send_textarea').val(result.newText).trigger('input');
-    }
-
-    // interrupt generation if the input was nothing but a command
-    if (message.length > 0 && result?.newText.length === 0) {
-        return true;
-    }
-
-    return result?.interrupt;
+    await executeSlashCommandsOnChatInput(message, {
+        clearChatInput: true,
+    });
+    return true;
 }
 
 export function sendSystemMessage(type, text, extra = {}) {
@@ -2450,6 +2439,14 @@ export function sendSystemMessage(type, text, extra = {}) {
     chat.push(newMessage);
     addOneMessage(newMessage);
     is_send_press = false;
+    if (type == system_message_types.SLASH_COMMANDS) {
+        const browser = new SlashCommandBrowser();
+        const spinner = document.querySelector('#chat .last_mes .custom-slashHelp');
+        const parent = spinner.parentElement;
+        spinner.remove();
+        browser.renderInto(parent);
+        browser.search.focus();
+    }
 }
 
 /**
@@ -2697,7 +2694,7 @@ class StreamingProcessor {
         let messageId = -1;
 
         if (this.type == 'impersonate') {
-            $('#send_textarea').val('').trigger('input');
+            $('#send_textarea').val('')[0].dispatchEvent(new Event('input', { bubbles:true }));
         }
         else {
             await saveReply(this.type, text, true);
@@ -2733,7 +2730,7 @@ class StreamingProcessor {
         }
 
         if (isImpersonate) {
-            $('#send_textarea').val(processedText).trigger('input');
+            $('#send_textarea').val(processedText)[0].dispatchEvent(new Event('input', { bubbles:true }));
         }
         else {
             let currentTime = new Date();
@@ -3080,7 +3077,7 @@ export async function Generate(type, { automatic_trigger, force_name2, quiet_pro
         const interruptedByCommand = await processCommands(String($('#send_textarea').val()));
 
         if (interruptedByCommand) {
-            //$("#send_textarea").val('').trigger('input');
+            //$("#send_textarea").val('')[0].dispatchEvent(new Event('input', { bubbles:true }));
             unblockGeneration(type);
             return Promise.resolve();
         }
@@ -3175,7 +3172,7 @@ export async function Generate(type, { automatic_trigger, force_name2, quiet_pro
     if (type !== 'regenerate' && type !== 'swipe' && type !== 'quiet' && !isImpersonate && !dryRun) {
         is_send_press = true;
         textareaText = String($('#send_textarea').val());
-        $('#send_textarea').val('').trigger('input');
+        $('#send_textarea').val('')[0].dispatchEvent(new Event('input', { bubbles:true }));
     } else {
         textareaText = '';
         if (chat.length && chat[chat.length - 1]['is_user']) {
@@ -4137,7 +4134,7 @@ export async function Generate(type, { automatic_trigger, force_name2, quiet_pro
 
         if (getMessage.length > 0) {
             if (isImpersonate) {
-                $('#send_textarea').val(getMessage).trigger('input');
+                $('#send_textarea').val(getMessage)[0].dispatchEvent(new Event('input', { bubbles:true }));
                 generatedPromptCache = '';
                 await eventSource.emit(event_types.IMPERSONATE_READY, getMessage);
             }
@@ -8477,19 +8474,123 @@ jQuery(async function () {
         toastr.success('Chat and settings saved.');
     }
 
-    registerSlashCommand('dupe', DupeChar, [], '– duplicates the currently selected character', true, true);
-    registerSlashCommand('api', connectAPISlash, [], `<span class="monospace">(${Object.keys(CONNECT_API_MAP).join(', ')})</span> – connect to an API`, true, true);
-    registerSlashCommand('impersonate', doImpersonate, ['imp'], '<span class="monospace">[prompt]</span> – calls an impersonation response, with an optional additional prompt', true, true);
-    registerSlashCommand('delchat', doDeleteChat, [], '– deletes the current chat', true, true);
-    registerSlashCommand('getchatname', doGetChatName, [], '– returns the name of the current chat file into the pipe', false, true);
-    registerSlashCommand('closechat', doCloseChat, [], '– closes the current chat', true, true);
-    registerSlashCommand('panels', doTogglePanels, ['togglepanels'], '– toggle UI panels on/off', true, true);
-    registerSlashCommand('forcesave', doForceSave, [], '– forces a save of the current chat and settings', true, true);
-    registerSlashCommand('instruct', selectInstructCallback, [], '<span class="monospace">(name)</span> – selects instruct mode preset by name. Gets the current instruct if no name is provided', true, true);
-    registerSlashCommand('instruct-on', enableInstructCallback, [], '– enables instruct mode', true, true);
-    registerSlashCommand('instruct-off', disableInstructCallback, [], '– disables instruct mode', true, true);
-    registerSlashCommand('context', selectContextCallback, [], '<span class="monospace">(name)</span> – selects context template by name. Gets the current template if no name is provided', true, true);
-    registerSlashCommand('chat-manager', () => $('#option_select_chat').trigger('click'), ['chat-history', 'manage-chats'], '– opens the chat manager for the current character/group', true, true);
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'dupe',
+        callback: DupeChar,
+        helpString: 'Duplicates the currently selected character.',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'api',
+        callback: connectAPISlash,
+        namedArgumentList: [],
+        unnamedArgumentList: [
+            new SlashCommandArgument(
+                'API to connect to',
+                [ARGUMENT_TYPE.STRING],
+                true,
+                false,
+                null,
+                Object.keys(CONNECT_API_MAP),
+            ),
+        ],
+        helpString: `
+            <div>
+                Connect to an API.
+            </div>
+            <div>
+                <strong>Available APIs:</strong>
+                <pre><code>${Object.keys(CONNECT_API_MAP).join(', ')}</code></pre>
+            </div>
+        `,
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'impersonate',
+        callback: doImpersonate,
+        aliases: ['imp'],
+        unnamedArgumentList: [
+            new SlashCommandArgument(
+                'prompt', [ARGUMENT_TYPE.STRING], false,
+            ),
+        ],
+        helpString: `
+            <div>
+                Calls an impersonation response, with an optional additional prompt.
+            </div>
+            <div>
+                <strong>Example:</strong>
+                <ul>
+                    <li>
+                        <pre><code class="language-stscript">/impersonate What is the meaning of life?</code></pre>
+                    </li>
+                </ul>
+            </div>
+        `,
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'delchat',
+        callback: doDeleteChat,
+        helpString: 'Deletes the current chat.',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'getchatname',
+        callback: doGetChatName,
+        returns: 'chat file name',
+        helpString: 'Returns the name of the current chat file into the pipe.',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'closechat',
+        callback: doCloseChat,
+        helpString: 'Closes the current chat.',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'panels',
+        callback: doTogglePanels,
+        aliases: ['togglepanels'],
+        helpString: 'Toggle UI panels on/off',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'forcesave',
+        callback: doForceSave,
+        helpString: 'Forces a save of the current chat and settings',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'instruct',
+        callback: selectInstructCallback,
+        returns: 'current preset',
+        namedArgumentList: [],
+        unnamedArgumentList: [
+            new SlashCommandArgument(
+                'name', [ARGUMENT_TYPE.STRING], false,
+            ),
+        ],
+        helpString: `
+            <div>
+                Selects instruct mode preset by name. Gets the current instruct if no name is provided.
+            </div>
+            <div>
+                <strong>Example:</strong>
+                <ul>
+                    <li>
+                        <pre><code class="language-stscript">/instruct creative</code></pre>
+                    </li>
+                </ul>
+            </div>
+        `,
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'instruct-on',
+        callback: enableInstructCallback,
+        helpString: 'Enables instruct mode.',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'instruct-off',
+        callback: disableInstructCallback,
+        helpString: 'Disables instruct mode',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'context',
+        callback: selectContextCallback,
+        returns: 'template name',
+        unnamedArgumentList: [
+            new SlashCommandArgument(
+                'name', [ARGUMENT_TYPE.STRING], false,
+            ),
+        ],
+        helpString: 'Selects context template by name. Gets the current template if no name is provided',
+    }));
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({ name: 'chat-manager',
+        callback: () => $('#option_select_chat').trigger('click'),
+        aliases: ['chat-history', 'manage-chats'],
+        helpString: 'Opens the chat manager for the current character/group.',
+    }));
 
     setTimeout(function () {
         $('#groupControlsToggle').trigger('click');
@@ -9867,11 +9968,22 @@ jQuery(async function () {
             streamingProcessor = null;
         }
         if (abortController) {
-            abortController.abort();
+            abortController.abort('Clicked stop button');
             hideStopButton();
         }
         eventSource.emit(event_types.GENERATION_STOPPED);
-        activateSendButtons();
+    });
+
+    $(document).on('click', '#form_sheld .stscript_continue', function () {
+        pauseScriptExecution();
+    });
+
+    $(document).on('click', '#form_sheld .stscript_pause', function () {
+        pauseScriptExecution();
+    });
+
+    $(document).on('click', '#form_sheld .stscript_stop', function () {
+        stopScriptExecution();
     });
 
     $('.drawer-toggle').on('click', function () {