Extension shell with `ellmer` chat
1-shell.RmdGoal
Phase 1 delivers a mountable dock_extension whose UI is
a shinychat panel and whose server constructs an
ellmer::Chat from the board’s llm_model
option. At the end of Phase 1 the package builds, the extension can be
added to any dock_board, and a user can hold a streaming,
cancellable conversation with the model. The assistant has no awareness
of the board yet — that is Phase 2.
Scope
In:
-
new_assistant_extension()constructor returning adock_extension. - A chat panel UI based on
shinychat::chat_mod_ui(). - An extension server that reads
llm_modelfrom the board options, invokes the constructor, and hands the resultingellmer::Chattoshinychat::chat_mod_server(). - A short, overridable default system prompt.
- A
stateshape that round-trips the system prompt and the full conversation throughblockr.dock’s ser/des. - A demo app under
inst/examples/that mounts the extension on adock_boardwith one or two starter blocks.
Out (deferred to later phases):
- Any awareness of board state, blocks, links, stacks.
- Tool registration (Phase 2 and 4).
- Dynamic board summary in the system prompt (Phase 5).
- Application-level persistence wiring. The extension is built so
history can round-trip through ser/des, but cross-session
save/restore (e.g. via
blockr.session) is not in this phase.
Architectural decisions
State shape mirrors the constructor signature.
blockr.dock’s extension ser/des serializes the extension
server’s state list entry-by-entry (with
reval_if) and on restore calls the constructor with those
entries spliced in as named arguments
(do.call(ctor_fun, c(payload, …)), see
blockr.dock/R/utils-serdes.R). The structural consequence:
every key in state must be a valid argument of
new_assistant_extension(), and conversely every constructor
argument that affects runtime behaviour should appear in
state if we want save/restore fidelity. This decision
shapes the rest of the doc.
Provider neutrality via
new_llm_model_option(). We never construct an
ellmer chat client directly. The board exposes an
llm_model board option whose value is a chat-constructor of
signature function(system_prompt = NULL, params = NULL)
returning an ellmer::Chat. blockr.core
guarantees a usable fallback (default_chat) so the
extension never needs to handle a missing backend.
The Chat client is constructed once per session.
Re-constructing on every llm_model option change would
force chat_mod_server to re-mount and lose its history. For
Phase 1 the option is read once at extension-server start; changing
models mid-session requires a session restart. Lifting this restriction
is future work.
Stream cancellation is delegated to shinychat,
not built ourselves. shinychat::chat_mod_server()
already instantiates an ellmer::stream_controller()
internally and wires input$chat_cancel to
controller$cancel(). The stop button surfaces from the
chat_mod_ui() chrome. The Phase 0 roadmap entry for a
“visible stop affordance” therefore costs nothing in this phase — we
only verify it works end-to-end in the demo.
Public API
new_assistant_extension(
system_prompt = NULL,
messages = NULL,
...
)-
system_prompt— character scalar orNULL. WhenNULL, the package default is used. A user-supplied string replaces the default verbatim; we do not concatenate. Callers who want to extend the default can read it via an unexporteddefault_system_prompt(). -
messages— optional list of recorded turns to seed the conversation with. The on-disk shape is the output ofellmer::contents_record()per turn (see Wire format below).NULLmeans start with an empty conversation. -
...— forwarded tonew_dock_extension()(notably foroptions =and thepkg =provenance argument).
Notably absent: a params argument. Model parameters
(temperature, max tokens, …) belong to the chat constructor, not to the
extension. A caller who wants different params should register a chat
constructor that bakes them in — either as the global
blockr_option("chat_function", …) or as a named alternative
selected via the board’s llm_model option. Having a second
params knob on the extension would create two ways to do
the same thing, with a precedence question if both are set, and would
force us to design a JSON ser/des for ellmer::params() (an
S7 object) just so it could round-trip through state.
Class: assistant_extension, inheriting from
dock_extension.
Extension UI
asst_ext_ui <- function(id, board, ...) {
shinychat::chat_mod_ui(NS(id, "chat"))
}chat_mod_ui() renders the message list, the input box
and the stop button. We namespace the inner id ("chat") so
later phases can add sibling outputs (token counter, debug panel)
without colliding with chat_mod_ui’s internal ids.
Extension server
asst_ext_srv <- function(id, board, update, ...) {
moduleServer(
id,
function(input, output, session) {
chat_ctor <- isolate(get_board_option_value("llm_model", session))
sys_prompt <- system_prompt %||% default_system_prompt()
client <- chat_ctor(system_prompt = sys_prompt)
if (length(messages)) {
client$set_turns(lapply(messages, ellmer::contents_replay))
}
mod <- shinychat::chat_mod_server("chat", client)
messages <- reactiveVal(messages %||% list())
record_new_turns <- function() {
recorded <- messages()
turns <- client$get_turns()
if (length(turns) > length(recorded)) {
new_idx <- seq.int(length(recorded) + 1L, length(turns))
messages(
c(recorded, lapply(turns[new_idx], ellmer::contents_record))
)
} else if (length(turns) < length(recorded)) {
messages(lapply(turns, ellmer::contents_record))
}
}
observeEvent(mod$last_input(), record_new_turns(), ignoreNULL = TRUE)
observeEvent(mod$last_turn(), record_new_turns(), ignoreNULL = TRUE)
list(
state = list(
system_prompt = sys_prompt,
messages = messages
)
)
}
)
}A few notes on the structure:
-
system_promptand themessagesargument are captured from thenew_assistant_extension()enclosing scope.system_promptstays a plain string throughout the session. Themessagesargument is consumed once to seedclient$set_turns()and the localreactiveValof the same name; after that the reactive form is what the rest of the server reads and writes. -
get_board_option_value("llm_model", session)isblockr.core’s accessor for board options inside a Shiny module. It returns the chat constructor with achat_nameattribute we can surface later in telemetry. The accessor reads a reactive value, so we wrap it inisolate()– the chat client is constructed once at module startup and must not re-fire if the option later changes. -
messagestracks the recorded form of the conversation. We append on each turn boundary (mod$last_inputfor the user’s submit,mod$last_turnfor the model’s response completion) rather than re-recording the whole list —contents_recordover a growing list of turns would be O(n²) over the conversation. Thelength(turns) < length(recorded)branch handles the edge case where something external (e.g.mod$clear()) shrinks the client’s turns; we re-sync rather than diverge. - The return shape
list(state = …)is the contract documented fordock_extensionservers (blockr.dock/R/ext-class.R). Every key instatematches a constructor argument by name — that is what makes ser/des coherent.
The live ellmer::Chat instance and the
chat_mod_server return (mod) are deliberately
not in state. They are runtime concerns (tool
registration in Phase 2, turn-completion observers in Phase 3), they are
not serializable, and they are not constructor arguments. Exposing them
externally — if later phases need it — is a separate decision from
ser/des.
Wire format for messages
Corrected. The “no opaque blobs” shape below was
written into state verbatim, and it does not
survive the board’s own JSON round trip. jsonlite reads
"version": 1 back as an integer, which the type-strict
check in ellmer::contents_replay() rejects, and typed props
come back mistyped — tokens as a list, and the
"NA" strings seen below failing S7 validation. Because the
replay runs in a board-server observer, a board saved after any chat
turn aborted the whole board on restore.
The record/replay pair is still what turns a Turn into
data, but the result is now stored as a single opaque
jsonlite::serializeJSON() string, which preserves R’s types
exactly and so round-trips through the board file unchanged. The saved
window is bounded by the blockr.chat_save_turns option,
read at save time, the raw provider response in json is
stripped, and truncation is trimmed to whole exchanges so a window never
opens on a tool result whose request fell outside it. The structure
below is therefore still the payload — it is simply no longer the thing
written to disk.
We use ellmer::contents_record() /
ellmer::contents_replay() as the round-trip pair. Output is
plain JSON — no opaque blobs — and the round-trip is lossless across all
turn shapes (text, tool calls, multi-content). For the simplest
exchange:
[
{ "version": 1, "class": "ellmer::UserTurn",
"props": {
"contents": [
{ "version": 1, "class": "ellmer::ContentText",
"props": { "text": "hi" } }
]
}
},
{ "version": 1, "class": "ellmer::AssistantTurn",
"props": {
"contents": [
{ "version": 1, "class": "ellmer::ContentText",
"props": { "text": "hello back" } }
],
"json": [], "tokens": ["NA","NA","NA"],
"cost": "NA", "duration": "NA"
}
}
]Verbosity here is the cost of lossless round-tripping — assistant
turns that carry tool calls (Phase 4) or token telemetry are faithfully
preserved. We deliberately do not roll our own
{role, content} shape: it would be lossy by Phase 4 and
would duplicate work ellmer already maintains.
Default system prompt
You are a helpful assistant embedded next to a blockr data analysis
board. In future versions you will be able to inspect and manipulate
the board; for now you can only talk to the user. Answer concisely.
Do not invent tool calls or claim to have changed the board — you
cannot, yet.
The “you cannot, yet” line is deliberate: in Phase 1 the model is board-blind, and without an explicit instruction it will happily hallucinate a CRUD vocabulary it does not have. Replacing this prompt becomes important in Phase 2 once real tools exist; we revisit the text there.
The prompt lives as a string constant in
R/system-prompt.R. A single constant is fine for now;
file-based prompts can come later when we start composing multiple
fragments (persona + board summary + tool catalogue).
Telemetry
ellmer::Chat exposes per-turn token accounting via
client$tokens() and client$last_turn()$tokens.
Phase 1 surfaces this as a small
verbatimTextOutput("tokens") beneath the chat:
input: 312 output: 84 total this turn: 396
The output is updated in an observer over mod$last_turn.
Cost estimation is out of scope for Phase 1 (it requires a per-model
price table we do not yet maintain); only raw token counts are
shown.
Token counts are not added to state. They are
recomputable from the recorded messages (the per-turn
tokens field is preserved by contents_record),
so exposing them separately would be redundant and would commit us to a
key that isn’t a constructor argument.
Demo app
inst/examples/populated-board/app.R mounts the extension
on a small board:
library(blockr.core)
library(blockr.dock)
library(blockr.assistant)
board <- new_dock_board(
blocks = list(new_dataset_block("iris")),
extensions = list(new_assistant_extension())
)
serve(board)Acceptance criteria for Phase 1:
- App launches without warnings against a board that has
llm_modelunset (i.e. thedefault_chatfallback is exercised). - A streaming response renders chunk-by-chunk in the chat panel.
- Clicking the stop button mid-stream stops the stream and leaves a
partial-turn entry in the history (
chat_mod_serverdefault behaviour). - The token line updates after each turn.
- Serialize a board after a three-turn conversation, restore it in a fresh session, confirm: (a) the conversation is visible in the chat panel, (b) the next user message correctly extends the restored history, (c) the system prompt survived the round-trip.
-
devtools::check()passes with 0/0/0.
Files added in this phase
-
R/extension.R—new_assistant_extension(),asst_ext_ui(),asst_ext_srv(). -
R/system-prompt.R—default_system_prompt(). -
R/utils.R—%||%and any small helpers shared with later phases. -
inst/examples/populated-board/app.R— demo app. -
tests/testthat/test-extension.R— smoke tests (is_dock_extension(), server runs undershiny::testServer()against adefault_chatmock, ser/des round-trip).
Known limitations carried into later phases
- Changing the
llm_modeloption mid-session has no effect — the client is captured at server start. Either documented as a known limitation or addressed by an explicit “reset chat” button later. - No board awareness: the chat cannot answer “what blocks do I have?” questions yet. This is the entire subject of Phase 2.
- No application-level persistence trigger. Phase 1 makes history serializable; what’s missing is the integration that actually saves and restores boards across sessions.