STscript Parser Rewrite (#1965)

* set isForced to true on input

* make floating auto-complete follow horizontal scrolling

* add callable closure vars

* changes to /let and /var for callable closures

* fix error message

* fix scope for closure arguments

* if should return the pipe result from closures

* use /run to call closures and no arguments on immediate closures

* throw exception from QRs window-function if no match

* when to show autocomplete vs info only

* autocomplete positioning

* autocomplete styling

* add theming to autocomplete (theme, dark, light)

* improve autocomplete show/hide logic and editor selection

* use blur tint color instead of chat tint color and use blur setting

* cleanup and docs

* use scope macros for QR args

* add enter to select autocomplete

* fix no executor found

* cleanup and comment

* fix alias list in help string

* fallback to empty string piped value if null or undefined

* fix typo

* blur textarea on ctrl+enter execute (and refocus after)

* stop executeSlashCommand if parser throws

* move /let and /var callbacks into functions

* switch textarea to monospace when value starts with slash

* add double pipe a pipe breaker

* fix /? slash

* remove some logging

* add "/:name" as shorthand for "/run name" after all

* move shit around

* fix error message

* use testRunShorthandEnd

* use parseQuotedValue and parseValue to determine name for "/:"

QR labels and set names can include spaces

* add some adjustments to make autocomplete work properly

some hint in there about "/:" would still be nice

* add autocomplete style  selector

* only strip quotes from subcommand if they are at both ends

* fix JSDoc

* escaping

* allow open quotes on dry run

* throwing shit at the wall for /: autocomplete

* escapes only for symbols

* clean up autocomplete

* improve performance

* fix scope macros

* remove unescaping of pipes

* fix macros in scope copy

* fix "/? slash"

* don't run parser for getNameAt if text has not changed

* fix options filter

* re-enable blur listener

* restore selection on non-replace select

* fix for escaping first character of value

* add support for {{pipe}} and {{var::}} closures

* add index support to var macro

* add scoped var macro to macro help

* more escape fixes

* reduce autocomplete render debounce

* cleanup

* restore old escape handling and parser flag for strict escaping

* fix "no match" autocomplete message

* add dummy commands for comments and parser flag

* fix type annotations

* somewhat safer macro replacements

* fix autocomplete select on blank / "no match"

* fix cutting off handled part in substitution

* add parser flag REPLACE_GETVAR

Replaces all {{getvar::}} and {{getglobalvar::}} macros with {{var::}}.
Inserts a series of command executors before the command with the macros that:
- save {{pipe}} to a var
- call /getvar or /getglobalvar to get the variable used in the macro
- call /let to save the retrieved variable
- return the saved {{pipe}} value

This helps to avoid double-substitutions when the var values contain text that could be interpreted as macros.

* remove old parser

* fix send on enter when no match

* deal with pipes in quoted values (loose escaping)

* add default parser flags to user settings

* allow quoted values in unnamed argument

* set parser flag without explicit state to "on"

* add click hint on parser error toast

* dirty more detailed cmd defs

* remove name from unnamed arg

* move autocomplete into class and floating with details

* replace jQuery's trigger('input') on #send_textarea with native events because jQuery does not dispatch the native event

* fix ctrl+space

* fix arrow navigation

* add comments

* fix pointer block

* add static fromProps

* fix up dummy commands

* migrate all commands to addCommandObject

* remove commented comment command

* fix alias in details

* add range as argument type

* switch to addCommandObject

* switch to addCommandObject

* fix height

* fix floating details position on left

* re-enable blur event

* use auto width for full details on floating autocomplete

* auto-size floating full details

* fix typo

* re-enable blur listener

* don't prevent enter when selected item is fully typed out

* add autocomplete details tooltips

* add language to slash command examples

* move makeItem into option and command and fix click select

* use autocomplete parts in /? slash

* fix alias formatting

* add language to slash command examples

* fix details position on initial input history

* small screen styles

* replace registerSlashCommand with detailed declarations

* put name on first line

* add missing returns

* fix missing comma

* fix alias display in autocomplete list

* remove args from help string

* move parser settings to its own section

* jsdoc

* hljs stscript lang

* add hljs to autocomplete help examples

* add missing import

* apply autocomplete colors to stscript codeblocks (hljs)

* add fromProps

* cache autocomplete elements

* towards generic autocomplete

* remove unused imports

* fix blanks

* add return types

* re-enable blur

* fix blank check

* Caption messages by id

* add aborting command execution

* fix return type

* fix chat input font reset

* add slash command progress indicator

* add missing return

* mark registerSlashCommand deprecated

* why??

* separate abort logic for commands

* remove parsing of quoted values from unnamed arg

* add adjustable autocomplete width

* revert stop button pulse

* add progress and pause/abort to QR editor

* add resize event on autocomplete width change

* add key= argument to all get vars

* refactoring

* introduce NamedArgumentAsignment

* add TODOs

* refactoring

* record start and end of named arg assignment

* refactoring

* prevent duplicate calls to show

* refactoring

* remove macro ac

* add secondary autocomplete and enum descriptions

* add syntax highlighting to QR editor

* add enum descriptions to /while

* add /let key=... to scope variable names

* add unnamed argument assignment class and unnamed argument splitting

* fix QR editor style

* remove dash before autocomplete help text

* add autocomplete for unnamed enums

* fix remaining dom after holding backslash

* fix for unnamed enums

* fix autocomplete for /parser-flag

* add parser-flag enum help

* fix type annotations

* fix autocomplete result for /:

* add colored autocomplete type icons

* collapse second line autocomplete help if empty

* mark optional named args in autocomplete

* fix when what

* remove duplicate debug buttons

* dispatch input on autocomplete select

* prevent grow from editor syntax layer

* add auto-adjust qr editor caret color

* remove text-shadow from autocomplete

* join value strings in /let and /var

* add /abort syntax highlight

* fix attempting secondary result when there is none

* rename settings headers and split autocomplete / stscript

* add parser flag tooltips

* add tooltips to chat width stops

* fix typo

* return clone of help item

* fix enum string

* don't make optional notice for autocomplete arguments smaller

* avoid scrollbar in chat input

* add rudimentary macro autocomplete

* strip macro from helptext

* finally remove closure delimiters around root

* cleanup

* fix index stuff for removed closure delimiters

* fix type hint

* add child commands to progress indicator

* include sub-separator in macro autocomplete

* remove all mentions of interruptsGeneration and purge

* remove unused imports

* fix syntax highlight with newline at end of input

* cleanup select pointer events

* coalesce onProgress call

* add regex to STscript syntax highlighting

* fix closure end

* fix autocomplete type icon alignment

* adjustments for small screens

* fix removing wrong element

* add missing "at=" arg to /sys, /comment, /sendas

* add font scale setting for autocomplete

* add target=_blank for parser flag links

* fix for searching enums

* remove REGEXP_MODE from hljs
just causes trouble

* fix autocomplete in closures

* fix typo

* fix type hint

* Get rid of scroll bar on load

* Add type hint for /send name argument. Fix 'at' types

* Add 'negative' arg hint to /sd command

* reenable blur event

* Allow /summarize to process any text

* Compact layout of script toggles

* Expand CSS by default

* fix double ranger indicator and adjust to narrow container

* make custom css input fill available vertical space

* reduce scroll lag

* use default cursor on scrollbar

* Clean-up module loading in index.html

* fix tab indent with hljs

---------

Co-authored-by: Cohee <18619528+Cohee1207@users.noreply.github.com>

public/scripts/extensions/memory/index.js

@@ -16,11 +16,14 @@ import {
     getMaxContextSize,
 } from '../../../script.js';
 import { is_group_generating, selected_group } from '../../group-chats.js';
-import { registerSlashCommand } from '../../slash-commands.js';
 import { loadMovingUIState } from '../../power-user.js';
 import { dragElement } from '../../RossAscends-mods.js';
 import { getTextTokens, getTokenCountAsync, tokenizers } from '../../tokenizers.js';
 import { debounce_timeout } from '../../constants.js';
+import { SlashCommandParser } from '../../slash-commands/SlashCommandParser.js';
+import { SlashCommand } from '../../slash-commands/SlashCommand.js';
+import { ARGUMENT_TYPE, SlashCommandArgument, SlashCommandNamedArgument } from '../../slash-commands/SlashCommandArgument.js';
+import { resolveVariable } from '../../variables.js';
 export { MODULE_NAME };
 
 const MODULE_NAME = '1_memory';
@@ -416,7 +419,7 @@ async function forceSummarizeChat() {
     console.log(`Skipping WIAN? ${skipWIAN}`);
     if (!context.chatId) {
         toastr.warning('No chat selected');
-        return;
+        return '';
     }
 
     toastr.info('Summarizing chat...', 'Please wait');
@@ -424,7 +427,42 @@ async function forceSummarizeChat() {
 
     if (!value) {
         toastr.warning('Failed to summarize chat');
-        return;
+        return '';
+    }
+
+    return value;
+}
+
+/**
+ * Callback for the summarize command.
+ * @param {object} args Command arguments
+ * @param {string} text Text to summarize
+ */
+async function summarizeCallback(args, text) {
+    text = text.trim();
+
+    // Using forceSummarizeChat to summarize the current chat
+    if (!text) {
+        return await forceSummarizeChat();
+    }
+
+    const source = args.source || extension_settings.memory.source;
+    const prompt = substituteParams((resolveVariable(args.prompt) || extension_settings.memory.prompt)?.replace(/{{words}}/gi, extension_settings.memory.promptWords));
+
+    try {
+        switch (source) {
+            case summary_sources.extras:
+                return await callExtrasSummarizeAPI(text);
+            case summary_sources.main:
+                return await generateRaw(text, '', false, false, prompt, extension_settings.memory.overrideResponseLength);
+            default:
+                toastr.warning('Invalid summarization source specified');
+                return '';
+        }
+    } catch (error) {
+        toastr.error(String(error), 'Failed to summarize text');
+        console.log(error);
+        return '';
     }
 }
 
@@ -668,37 +706,18 @@ async function summarizeChatExtras(context) {
     // perform the summarization API call
     try {
         inApiCall = true;
-        const url = new URL(getApiUrl());
-        url.pathname = '/api/summarize';
+        const summary = await callExtrasSummarizeAPI(resultingString);
+        const newContext = getContext();
 
-        const apiResult = await doExtrasFetch(url, {
-            method: 'POST',
-            headers: {
-                'Content-Type': 'application/json',
-                'Bypass-Tunnel-Reminder': 'bypass',
-            },
-            body: JSON.stringify({
-                text: resultingString,
-                params: {},
-            }),
-        });
-
-        if (apiResult.ok) {
-            const data = await apiResult.json();
-            const summary = data.summary;
-
-            const newContext = getContext();
-
-            // something changed during summarization request
-            if (newContext.groupId !== context.groupId
-                || newContext.chatId !== context.chatId
-                || (!newContext.groupId && (newContext.characterId !== context.characterId))) {
-                console.log('Context changed, summary discarded');
-                return;
-            }
-
-            setMemoryContext(summary, true);
+        // something changed during summarization request
+        if (newContext.groupId !== context.groupId
+            || newContext.chatId !== context.chatId
+            || (!newContext.groupId && (newContext.characterId !== context.characterId))) {
+            console.log('Context changed, summary discarded');
+            return;
         }
+
+        setMemoryContext(summary, true);
     }
     catch (error) {
         console.log(error);
@@ -708,6 +727,40 @@ async function summarizeChatExtras(context) {
     }
 }
 
+/**
+ * Call the Extras API to summarize the provided text.
+ * @param {string} text Text to summarize
+ * @returns {Promise<string>} Summarized text
+ */
+async function callExtrasSummarizeAPI(text) {
+    if (!modules.includes('summarize')) {
+        throw new Error('Summarize module is not enabled in Extras API');
+    }
+
+    const url = new URL(getApiUrl());
+    url.pathname = '/api/summarize';
+
+    const apiResult = await doExtrasFetch(url, {
+        method: 'POST',
+        headers: {
+            'Content-Type': 'application/json',
+            'Bypass-Tunnel-Reminder': 'bypass',
+        },
+        body: JSON.stringify({
+            text: text,
+            params: {},
+        }),
+    });
+
+    if (apiResult.ok) {
+        const data = await apiResult.json();
+        const summary = data.summary;
+        return summary;
+    }
+
+    throw new Error('Extras API call failed');
+}
+
 function onMemoryRestoreClick() {
     const context = getContext();
     const content = $('#memory_contents').val();
@@ -865,5 +918,16 @@ jQuery(async function () {
     eventSource.on(event_types.MESSAGE_EDITED, onChatEvent);
     eventSource.on(event_types.MESSAGE_SWIPED, onChatEvent);
     eventSource.on(event_types.CHAT_CHANGED, onChatEvent);
-    registerSlashCommand('summarize', forceSummarizeChat, [], '– forces the summarization of the current chat using the Main API', true, true);
+    SlashCommandParser.addCommandObject(SlashCommand.fromProps({
+        name: 'summarize',
+        callback: summarizeCallback,
+        namedArgumentList: [
+            new SlashCommandNamedArgument('source', 'API to use for summarization', [ARGUMENT_TYPE.STRING], false, false, '', ['main', 'extras']),
+            new SlashCommandNamedArgument('prompt', 'prompt to use for summarization', [ARGUMENT_TYPE.STRING, ARGUMENT_TYPE.VARIABLE_NAME], false, false, ''),
+        ],
+        unnamedArgumentList: [
+            new SlashCommandArgument('text to summarize', [ARGUMENT_TYPE.STRING], false, false, ''),
+        ],
+        helpString: 'Summarizes the given text. If no text is provided, the current chat will be summarized. Can specify the source and the prompt to use.',
+    }));
 });