Let us compare it with a continued fraction approximation. For example, using the (first) sequence line of On-line Encyclopedia of Integer Sequences (OEIS) A001203 produces π with precision 100:
Next, we show the Pareto principle manifestation of for the continued fraction terms. First we observe that the terms a distribution similar to Benford’s law :
This document (notebook) describes three ways of making mazes (or labyrinths) using graphs. The first two are based on rectangular grids; the third on a hexagonal grid.
All computational graph features discussed here are provided by the Graph functionalities of Wolfram Language.
TL;DR
Just see the maze pictures below. (And try to solve the mazes.)
Procedure outline
The first maze is made by a simple procedure which is actually some sort of cheating:
A regular rectangular grid graph is generated with random weights associated with its edges.
The (minimum) spanning tree for that graph is found.
That tree is plotted with exaggeratedly large vertices and edges, so the graph plot looks like a maze.
This is “the cheat” — the maze walls are not given by the graph.
The second maze is made “properly”:
Two interlacing regular rectangular grid graphs are created.
The second one has one less row and one less column than the first.
The vertex coordinates of the second graph are at the centers of the rectangles of the first graph.
The first graph provides the maze walls; the second graph is used to make paths through the maze.
In other words, to create a solvable maze.
Again, random weights are assigned to edges of the second graph, and a minimum spanning tree is found.
There is a convenient formula that allows using the spanning tree edges to remove edges from the first graph.
In that way, a proper maze is derived.
The third maze is again made “properly” using the procedure above with two modifications:
Two interlacing regular grid graphs are created: one over a hexagonal grid, the other over a triangular grid.
The hexagonal grid graph provides the maze walls; the triangular grid graph provides the maze paths.
Since the formula for wall removal is hard to derive, a more robust and universal method based on nearest neighbors is used.
Simple Maze
In this section, we create a simple, “cheating” maze.
Remark: The steps are easy to follow, given the procedure outlined in the introduction.
The “maze” above looks like a maze because the vertices and edges are rectangular with matching sizes, and they are thicker than the spaces between them. In other words, we are cheating.
To make that cheating construction clearer, let us plot the shortest path from the bottom left to the top right and color the edges in pink (salmon) and the vertices in red:
finder = Nearest[Thread[GraphEmbedding[g1] -> VertexList[g1]]]
Take a maze edge and its vertex points:
e = First@EdgeList[mazePath];
aMazePathCoords = Association@Thread[VertexList[mazePath] -> GraphEmbedding[mazePath]];
points = List @@ (e /. aMazePathCoords)
Find the edge’s midpoint and the nearest wall-graph vertices:
Program a few versions of circle chords based visualization routines.
Called chord trail plots below.
Marvel at chord trail plots for larger moduli.
Make multiple collections of them.
Look into number of primitive roots distributions.
Consider making animations of the collections.
The animations should not be “chaotic” — they should have some inherent visual flow in them.
Consider different ways of sorting chord trail plots.
Using number theoretic arguments.
Yeah, would be nice, but requires too much head scratching and LLM-ing.
Convert plots to images and sort them.
Some might say that that is a “brute force” application.
Simple image sort does not work.
Latent Semantic Analysis (LSA) application.
After failing to sort the chord trail image collections by “simple” means, the idea applying LSA came to mind.
LSA being, of course, a favorite technique that was applied to sorting images multiple times in the past, in different contexts, [AAn1, AAn3, AAn4, AAn5, AAv3].
The function ChordTrails can be generalized to take a (pre-computed) chords argument. Here is an example of chords plot that connects integers that are modular inverses of each other:
In order to sort the plots we find dimension reduction basis representation of the corresponding images and sort using that representation. For more details see “Re-exploring the structure of Chinese character images”, [AAn1].
The Doomsday Clock is a symbolic timepiece maintained by the Bulletin of the Atomic Scientists (BAS) since 1947. It represents how close humanity is perceived to be to global catastrophe, primarily nuclear war but also including climate change and biological threats. The clock’s hands are set annually to reflect the current state of global security; midnight signifies theoretical doomsday.
We take text data from the past announcements, and extract the Doomsday Clock reading statements.
Evolution of Doomsday Clock times
We extract relevant Doomsday Clock timeline data from the corresponding Wikipedia page.
(Instead of using a page from BAS.)
We show how timeline data from that Wikipedia page can be processed with “standard” Wolfram Language (WL) functions and with LLMs.
The result plot shows the evolution of the minutes to midnight.
The plot could show trends, highlighting significant global events that influenced the clock setting.
Hence, we put in informative callouts and tooltips.
The data extraction and visualization in the notebook serve educational purposes or provide insights into historical trends of global threats as perceived by experts. We try to make the ingestion and processing code universal and robust, suitable for multiple evaluations now or in the (near) future.
Remark: Keep in mind that the Doomsday Clock is a metaphor and its settings are not just data points but reflections of complex global dynamics (by certain experts and a board of sponsors.)
By observing the (plain) text of that page we see the Doomsday Clock time setting can be extracted from the sentence(s) that begin with the following phrase:
startPhrase = "Bulletin of the Atomic Scientists";
sentence = Select[Map[StringTrim, StringSplit[txtEN, "\n"]], StringStartsQ[#, startPhrase] &] // First
(*"Bulletin of the Atomic Scientists, with a clock reading 90 seconds to midnight"*)
Remark: The EBNF grammar above can be obtained with LLMs using a suitable prompt with example sentences. (We do not discuss that approach further in this notebook.)
Here the parsing functions are generated from the EBNF string above:
ClearAll["p*"]
res = GenerateParsersFromEBNF[ParseToEBNFTokens[ebnf]];
res // LeafCount
(*375*)
We must redefine the parser pANY (corresponding to the EBNF rule “”) in order to prevent pANY of gobbling the word “clock” and in that way making the parser pOPENING fail.
54jfnd 9y2f clock is reading 46 second to midnight
clock is reading 900 minutes to midnight
clock is reading 955 second to midnight
clock reading 224 minute to midnight
clock reading 410 minute to midnight
jdsf5at clock reading 488 seconds to midnight
Here is a verification table with correct- and incorrect spellings:
lsPhrases = {
"doomsday clock is reading 2 seconds to midnight",
"dooms day cloc is readding 2 minute and 22 sekonds to mildnight"};
ParsingTestTable[pCLOCKREADING, lsPhrases, "Layout" -> "Vertical"]
Parsing of numeric word forms
One way to make the parsing more robust is to implement the ability to parse integer names (or numeric word forms) not just integers.
There are two parsing results for “fifty seven”, because pWordedInteger is defined with p10s⊗pUpTo10⊕p10s… . This can be remedied by using ParseJust or ParseShortest:
Let us try the new parser using integer names for the clock time:
str = "the doomsday clock is reading two minutes and forty five seconds to midnight";
pTOP[ToTokens@str]
(*{{{}, {"Minutes" -> 2, "Seconds" -> 45}}}*)
Enhance with LLM parsing
There are multiple ways to employ LLMs for extracting “clock readings” from arbitrary statements for Doomsday Clock readings, readouts, and measures. Here we use LLM few-shot training:
flop = LLMExampleFunction[{
"the doomsday clock is reading two minutes and forty five seconds to midnight" -> "{\"Minutes\":2, \"Seconds\": 45}",
"the clock of the doomsday gives 92 seconds to midnight" -> "{\"Minutes\":0, \"Seconds\": 92}",
"The bulletin atomic scienist maybe is set to a minute an 3 seconds." -> "{\"Minutes\":1, \"Seconds\": 3}"
}, "JSON"]
Here is an example invocation:
flop["Maybe the doomsday watch is at 23:58:03"]
(*{"Minutes" -> 1, "Seconds" -> 57}*)
The following function combines the parsing with the grammar and the LLM example function — the latter is used for fallback parsing:
Here is the application of the combine function above over a certain “random” Doomsday Clock statement:
s = "You know, sort of, that dooms-day watch is 1 and half minute be... before the big boom. (Of doom...)";
GetClockReading[s]
(*<|"Minutes" -> 1, "Seconds" -> 30|>*)
Remark: The same type of robust grammar-and-LLM combination is explained in more detail in the video “Robust LLM pipelines (Mathematica, Python, Raku)”, [AAv1]. (See, also, the corresponding notebook [AAn1].)
Timeline
In this section we extract Doomsday Clock timeline data and make a corresponding plot.
Here we get the Doomsday Clock timeline table from that page in JSON format using an LLM:
res =
LLMSynthesize[{
"Give the time table of the doomsday clock as a time series that is a JSON array.",
"Each element of the array is a dictionary with keys 'Year', 'MinutesToMidnight', 'Time', 'Description'.",
txtWk,
LLMPrompt["NothingElse"]["JSON"]
},
LLMEvaluator -> LLMConfiguration[<|"Provider" -> "OpenAI", "Model" -> "gpt-4o", "Temperature" -> 0.4, "MaxTokens" -> 5096|>]
]
(*"```json[{\"Year\": 1947, \"MinutesToMidnight\": 7, \"Time\": \"23:53\", \"Description\": \"The initial setting of the Doomsday Clock.\"},{\"Year\": 1949, \"MinutesToMidnight\": 3, \"Time\": \"23:57\", \"Description\": \"The Soviet Union tests its first atomic bomb, officially starting the nuclear arms race.\"}, ... *)
Post process the LLM result:
res2 = ToString[res, CharacterEncoding -> "UTF-8"];
res3 = StringReplace[res2, {"```json", "```"} -> ""];
res4 = ImportString[res3, "JSON"];
res4[[1 ;; 3]]
(*{{"Year" -> 1947, "MinutesToMidnight" -> 7, "Time" -> "23:53", "Description" -> "The initial setting of the Doomsday Clock."}, {"Year" -> 1949, "MinutesToMidnight" -> 3, "Time" -> "23:57", "Description" -> "The Soviet Union tests its first atomic bomb, officially starting the nuclear arms race."}, {"Year" -> 1953, "MinutesToMidnight" -> 2, "Time" -> "23:58", "Description" -> "The United States and the Soviet Union test thermonuclear devices, marking the closest approach to midnight until 2020."}}*)
Make a dataset with the additional column “Date” (having date-objects):
Remark: The LLM derived descriptions above are shorter than the descriptions in the column “Reason” of the dataset obtained parsing the page data. For the plot tooltips below we use the latter.
Timeline plot
In order to have informative Doomsday Clock evolution plot we obtain and partition dataset’s time series into step-function pairs:
Remark: By hovering with the mouse over the black points the corresponding descriptions can be seen. We considered using clock-gauges as tooltips, but showing clock-settings reasons is more informative.
As expected, parsing, plotting, or otherwise processing the Doomsday Clock settings and statements are excellent didactic subjects for textual analysis (or parsing) and temporal data visualization. The visualization could serve educational purposes or provide insights into historical trends of global threats as perceived by experts. (Remember, the clock’s settings are not just data points but reflections of complex global dynamics.)
One possible application of the code in this notebook is to make a “web service“ that gives clock images with Doomsday Clock readings. For example, click on this button:
… or “Making Robust LLM Computational Pipelines from Software Engineering Perspective”
Abstract
Large Language Models (LLMs) are powerful tools with diverse capabilities, but from Software Engineering (SE) Point Of View (POV) they are unpredictable and slow. In this presentation we consider five ways to make more robust SE pipelines that include LLMs. We also consider a general methodological workflow for utilizing LLMs in “every day practice.”
Here are the five approaches we consider:
DSL for configuration-execution-conversion
Infrastructural, language-design level solution
Detailed, well crafted prompts
AKA “Prompt engineering”
Few-shot training with examples
Via a Question Answering System (QAS) and code templates
Grammar-LLM chain of responsibility
Testings with data types and shapes over multiple LLM results
Compared to constructing SE pipelines, Literate Programming (LP) offers a dual or alternative way to use LLMs. For that it needs support and facilitation of:
Convenient LLM interaction (or chatting)
Document execution (weaving and tangling)
The discussed LLM workflows methodology is supported in Python, Raku, Wolfram Language (WL). The support in R is done via Python (with “reticulate”, [TKp1].)
The presentation includes multiple examples and showcases.
Modeling of the LLM utilization process is hinted but not discussed.
All systematic approaches of unfolding and refining workflows based on LLM functions, will include several decision points and iterations to ensure satisfactory results.
This flowchart outlines such a systematic approach:
We do not examine the data source and we do not want to reason too much about the data using the stats. We started this notebook by just wanting to make the bubble charts (both 2D and 3D.) Nevertheless, we are tempted to say and justify statements like:
Here is the Pareto principle plot of for the number of created (or renamed) programming languages per creator (using the WFR function ParetoPrinciplePlot):
We can see that ≈25% of the creators correspond to ≈50% of the languages.
Popularity
Obviously, programmers can and do use more than one programming language. Nevertheless, it is interesting to see the Pareto principle plot for the languages “mind share” based on the number of users estimates.
Remark: Note that we “cheat” by adding 1 before taking the logarithms.
We obtain the tables of correlations plots using the newly introduced, experimental PairwiseListPlot. If we remove the rows with zeroes some of the correlations become more obvious. Here is the corresponding tab view of the two correlation tables:
Remark: In the 3D bubble chart plot “Mathematica” and “Wolfram Language” are easier to discern.
Second system effect traces
In this section we try — and fail — to demonstrate that the more programming languages a team of creators makes the less successful those languages are. (Maybe, because they are more cumbersome and suffer the Second system effect?)
Remark: This section is mostly made “for fun.” It is not true that each sets of languages per creators team is made of comparable languages. For example, complementary languages can be in the same set. (See, HTTP, HTML, URL.) Some sets are just made of the same language but with different names. (See, Perl 6 and Raku, and Mathematica and Wolfram Language.) Also, older languages would have the First mover advantage.
In this blog post (corresponding to this notebook) we discuss data extraction techniques from the Web site Mediazona that tracks the Russian casualties in Ukraine. See [MZ1].
Since we did not find a public source code (or data) repository (like GitHub) of the data, we extract the data directly from the web site [MZ1]. We can use both (i) image processing and (ii) web browser automation. But since we consider the latter to be both time consuming and unreliable to reproduce, in this notebook we consider only image processing (combined with AI vision.)
We did not “harvest” all types of data from Mediazona, only the casualties per week and day for all troops. (Which we see as most important.)
This notebook is intentionally kept to be only “technical know-how”, without further data analysis, or correlation confirmations with other publications, or model applications, etc. We plan to do analysis and modeling in other notebooks/articles. (Using data from Mediazona and other sources.)
Remark: At the time of programming the extractions of this notebook, (2023-11-29), Midiazona, [MZ1], says that the Russian casualties it presents are corroborated by publicly available data as of 17 November, 2023.
Remark: Mediazona is Anti Putinist, [Wk1], and (judging from its publications) it is pro-Ukraine and pro-West.
Similar other data sources
Here is a couple of other data sources with similar intent or mission:
Daily data extraction from weekly bar chart tooltips
Additional comments and remarks
The second and fourth sections have subsections that outline the corresponding procedures.
Images with data
At first we got two images from [MZ1]: one for casualties per week and one for casualties per day. (For all troops.)
Then in order to extract more faithful daily casualties data we took ≈90 screenshots of the weekly casualties bar chart at [MZ1], each screenshot with a tooltip shown for a different week.
Casualties per week
Casualties per day
Screenshots of weekly bar chart with tooltips
In order to get more faithful data readings of the daily casualties multiple (≈90) screenshots were taken of the weekly casualties bar chart, each of the screenshots having a tooltip table of one (unique) bar. It took ≈15 minutes to take those screenshots. They can be obtained from this Google Drive link.
Here is how one of them looks like:
Number of days and number weeks
Here is the number of weeks we expect to see in the “Casualties per week” plot:
The extracted data has to be rescaled to match the reported data. (We can see we have to “calibrate” the extracted data over a few points of the real data.)
Here we remake the plot above to include characteristic points we can use the calibration:
Remark: The reported total number of casualties can be seen in the original weekly casualties screenshot above.
Daily data extraction from daily bar chart
Daily casualties extraction is not that easy with technique applied to the weekly casualties plot. One of the reasons is that the daily casualties plot is also a user input interface(on that web page).
Since we want to get daily data for calibration of (generalized) Lanchester law models we can simply extrapolate the weekly data with daily averages. We can also over-impose in some way the two images (or plots) in order to convince ourselves that we have a faithful enough interpolation.
Nevertheless, more faithful daily data can be obtained by image- and LLM processing the tooltips of the weekly casualties chart. (See the next section.)
Daily data extraction from weekly bar chart tooltips
Procedure
Here is the procedure outline:
Take multiple screenshots of the weekly casualties bar chart
A screenshot for each week with the corresponding tooltip shown
Make sure all screenshots have the same size (or nearly the same size)
E.g. take “window screenshots”
≈90 screenshots can be taken within 15 minutes
Crop the screenshots appropriately
In order to get the tooltip tables only for each screenshot:
Here we do batch AI vision application, [AAn1], using an appropriate prompt:
h = 11;
AbsoluteTiming[
lsImgTableJSONs =
Table[(
Echo[Style[{i, i + (h - 1)}, Purple, Bold], "Span:"];
t =
LLMVisionSynthesize[{
"Get the 1) week span, 2) total casualties 3) count per day from the image.\n",
"Give the result as a JSON record with keys 'week_span', 'total_casualties', and 'count_per_day'.\n",
"Here is example of the JSON record for each image:{\"week_span\": \"10 Mar 2022 - 16 Mar 2022\",\"total_casualties\": 462,\"count_per_day\": {\"10 Mar\": 50,\"11 Mar\": 64,\"12 Mar\": 98,\"13 Mar\": 65,\"14 Mar\": 76,\"15 Mar\": 57,\"16 Mar\": 52}}",
LLMPrompt["NothingElse"]["JSON"]
},
Take[lsImgTables, {i, UpTo[i + (h - 1)]}],
"MaxTokens" -> 1200, "Temperature" -> 0.1];
Echo[t, "OCR:"];
t
),
{i, 1, Length[lsImgs], h}];
]
The two data extraction procedures agree very well over the extracted totals of casualties.
(Also good agreement with the “official” published total — approximately $3$‰ and $0.1$‰ respectively.)
LLMVision package
The function LLMVisionSynthesize used above is from the package “LLMVision.m”, [AAp1, AAn1]. One of the primary reasons to develop the package “LLMvision.m” was to use it in workflows like those above — extracting data from different sources in order to do war simulations.
Remark: In the section above LLMVisionSynthesize uses Base64 conversion of images. OpenAI’s Vision documentation advices to use URLs instead of Base64 images in long conversations.
Why apply image transformations when using AI vision?
One can ask:
Why do certain image transformations, or other image preprocessing, if we are using AI vision functionalities?
Can’t we just apply the AI?!
There are multiple reasons for preprocessing the images that are on different conceptual and operational levels:
We want to be able to use the same workflow but with different OCR algorithms that are “smaller” and “less AI”
Images having only the information to be extracted produce more reliable results
This obvious when OCR functions are used (like TextRecognize)
Less prompt engineering would be needed with AI-vision (most likely)
It is much cheaper — both computationally and money-wise — to use some smaller images for processed conveniently
Remark: OpenAI’s vision documentation discusses the money costs, preferred image formats, and reliability — see this “Limitations” section.
In the fall of 2023 OpenAI introduced the image vision model “gpt-4-vision-preview”, [OAIb1].
The model “gpt-4-vision-preview” represents a significant enhancement to the GPT-4 model, providing developers and AI enthusiasts with a more versatile tool capable of interpreting and narrating images alongside text. This development opens up new possibilities for creative and practical applications of AI in various fields.
For example, consider the following Wolfram Language (WL), developer-centric applications:
Narration of UML diagrams
Code generation from narrated (and suitably tweaked) narrations of architecture diagrams and charts
Generating presentation content draft from slide images
Extracting information from technical plots
etc.
A more diverse set of the applications would be:
Dental X-ray images narration
Security or baby camera footage narration
How many people or cars are seen, etc.
Transportation trucks content descriptions
Wood logs, alligators, boxes, etc.
Web page visible elements descriptions
Top menu, biggest image seen, etc.
Creation of recommender systems for image collections
Based on both image features and image descriptions
etc.
As a first concrete example, consider the following image that fable-dramatizes the name “Wolfram” (https://i.imgur.com/UIIKK9w.jpg):
LLMVisionSynthesize["Describe very concisely the image","https://i.imgur.com/UIIKK9w.jpg","MaxTokens"->600]
You are looking at a stylized black and white illustration of a wolf and a ram running side by side among a forest setting, with a group of sheep in the background. The image has an oval shape.
Remark: In this notebook Mathematica and Wolfram Language (WL) are used as synonyms.
Remark: This notebook is the WL version of the notebook “AI vision via Raku”, [AA3].
Ways to use with WL
There are five ways to utilize image interpretation (or vision) services in WL:
Remark: The model “gpt-4-vision-preview” is given as a “chat completion model” , therefore, in this document we consider it to be a Large Language Model (LLM).
Packages and paclets
Here we load WL package used below, [AAp1, AAp2, AAp3]:
Remark: The package LLMVision is “temporary” – It should be made into a Wolfram repository paclet, or (much better) its functionalities should be included in the “LLMFunctions” framework, [WRIp1].
Images
Here are the links to all images used in this document:
tblImgs ={{Row[{"Wolf and ram running together in forest"}],Row[{"https://i.imgur.com/UIIKK9w.jpg",""}]},{Row[{"LLM"," ","functionalities"," ","mind-map",""}],Row[{"https://i.imgur.com/kcUcWnql.jpg",""}]},{Row[{"Single"," ","sightseer",""}],Row[{"https://i.imgur.com/LEGfCeql.jpg",""}]},{Row[{"Three"," ","hunters",""}],Row[{"https://raw.githubusercontent.com/antononcube/Raku-WWW-OpenAI/main/resources/ThreeHunters.jpg",""}]},{Row[{"Cyber"," ","Week"," ","Spending"," ","Set"," ","to"," ","Hit"," ","New"," ","Highs"," ","in"," ","2023",""}],Row[{"https://cdn.statcdn.com/Infographic/images/normal/7045.jpeg",""}]}};tblImgs =Map[Append[#[[1 ;; 1]],Hyperlink[#[[-1,1,1]]]] &, tblImgs];TableForm[tblImgs,TableHeadings->{None,{"Name","Link"}}]/.{ButtonBox[n_,BaseStyle->"Hyperlink",ButtonData->{ URL[u_],None}] :> Hyperlink[n, URL[u]]}
Here is the structure of the rest of the document:
LLM synthesizing
… using multiple image specs of different kind.
LLM functions
… workflows over technical plots.
Dedicated notebook cells
… just excuses why they are not programmed yet.
Combinations (fairytale generation)
… Multi-modal applications for replacing creative types.
Conclusions and leftover comments
… frustrations untold.
LLM synthesizing
The simplest way to use the OpenAI’s vision service is through the function LLMVisionSynthesize of the package “LLMVision”, [AAp1]. (Already demoed in the introduction.)
If the function LLMVisionSynthesize is given a list of images, a textual result corresponding to those images is returned. The argument “images” is a list of image URLs, image file names, or image Base64 representations. (Any combination of those element types can be specified.)
Before demonstrating the vision functionality below we first obtain and show a couple of images.
OpenAI’s vision endpoint accepts POST specs that have image URLs or images converted into Base64 strings. When we use the LLMVisionSynthesize function and provide a file name under the “images” argument, the Base64 conversion is automatically applied to that file. Here is an example of how we apply Base64 conversion to the image from a given file path:
Here is an image narration example with the two images above, again, one specified with a URL, the other with a file path:
LLMVisionSynthesize["Give concise descriptions of the images.",{"https://i.imgur.com/LEGfCeql.jpg",$HomeDirectory <> "/Downloads/ThreeHunters.jpg"},"MaxTokens"->600]1. The first image depicts a single raccoon perched on a tree branch, surrounded by a plethora of vibrant, colorful butterflies in various shades of blue, orange, and other colors, set against a lush, multicolored foliage background.
2. The second image shows three raccoons sitting together on a tree branch in a forest setting, with a warm, glowing light illuminating the scene from behind. The forest is teeming with butterflies, matching the one in the first image, creating a sense of continuity and shared environment between the two scenes.
Description of a mind-map
Here is an application that should be more appealing to WL-developers – getting a description of a technical diagram or flowchart. Well, in this case, it is a mind-map from [AA2]:
Import[URL["https://i.imgur.com/kcUcWnql.jpeg"]]
Here are get the vision model description of the mind-map above (and place the output in Markdown format):
mmDescr = LLMVisionSynthesize["How many branches this mind-map has? Describe each branch separately. Use relevant emoji prefixes.","https://imgur.com/kcUcWnq.jpeg","MaxTokens"->900]
This mind map has four primary branches, each diverging from a \
central node labeled "LLM functionalities." I will describe each one \
using relevant emoji prefixes:
1. 🖼️ **DALL-E** branch is in yellow and represents an access point to \
the DALL-E service, likely a reference to a Large Language Model \
(LLM) with image generation capabilities.
2. 🤖 **ChatGPT** branch in pink is associated with the ChatGPT \
service, suggesting it's a conversational LLM branch. There are two \
sub-branches:
- **LLM prompts** indicates a focus on the prompts used to \
communicate with LLMs.
- **Notebook-wide chats** suggests a feature or functionality for \
conducting chats across an entire notebook environment.
3. 💬 **LLM chat objects** branch in purple implies that there are \
objects specifically designed for chat interactions within LLM \
services.
4. ✍️ **LLM functions** branch in green seems to represent various \
functional aspects or capabilities of LLMs, with a sub-branch:
- **Chatbooks** which may indicate a feature or tool related to \
managing or organizing chat conversations as books or records.
Converting descriptions to diagrams
Here from the obtained description we request a (new) Mermaid-JS diagram to be generated:
mmdChart = LLMSynthesize[{LLMPrompt["CodeWriter"],"Make the corresponding Mermaid-JS diagram code for the following description. Give the code only, without Markdown symbols.", mmDescr}]
graph TB
center[LLM functionalities]
center --> dalle[DALL-E]
center --> chat[ChatGPT]
center --> chatobj[LLM chat objects]
center --> functions[LLM functions]
chat --> prompts[LLM prompts]
chat --> notebook[Notebook-wide chats]
functions --> chatbooks[Chatbooks]
Here is a diagram made with the Mermaid-JS spec obtained above using the resource function of “MermaidInk”, [AAf1]:
ResourceFunction["MermaidInk"][mmdChart]
Below is given an instance of one of the better LLM results for making a Mermaid-JS diagram over the “vision-derived” mind-map description.
ResourceFunction["MermaidInk"]["
graph
TBA[LLM services access] --> B[DALL-E]
A --> C[ChatGPT]
A --> D[PaLM]
A --> E[LLM chat objects]
A --> F[Chatbooks]
B -->|related to| G[DALL-E AI system]
C -->|associated with| H[ChatGPT]
D -->|related to| I[PaLM model]
E -->|part of| J[chat-related objects/functionalities]
F -->|implies| K[Feature or application related to chatbooks]
"]
Code generation from image descriptions
Here is an example of code generation based on the “vision derived” mind-map description above:
LLMSynthesize[{LLMPrompt["CodeWriter"],"Generate the Mathematica code of a graph that corresponds to the description:\n", mmDescr}]
Consider another “serious” example – that of analyzing chess play positions. Here we get a chess position using the paclet “Chess”, [WRIp3]:
Here we describe it with “AI vision”:
LLMVisionSynthesize["Describe the position.",Image[b2],"MaxTokens"->1000,"Temperature"->0.05]
This is a chess position from a game in progress. Here's the \
description of the position by algebraic notation for each piece:
White pieces:
- King (K) on c1
- Queen (Q) on e2
- Rooks (R) on h1 and a1
- Bishops (B) on e3 and f1
- Knights (N) on g4 and e2
- Pawns (P) on a2, b2, c4, d4, f2, g2, and h2
Black pieces:
- King (K) on e8
- Queen (Q) on e7
- Rooks (R) on h8 and a8
- Bishops (B) on f5 and g7
- Knights (N) on c6 and f6
- Pawns (P) on a7, b7, c7, d7, f7, g7, and h7
It's Black's turn to move. The position suggests an ongoing middle \
game with both sides having developed most of their pieces. White has \
castled queenside, while Black has not yet castled. The white knight \
on g4 is putting pressure on the black knight on f6 and the pawn on \
h7. The black bishop on f5 is active and could become a strong piece \
depending on the continuation of the game.
Remark: In our few experiments with these kind of image narrations, a fair amount of the individual pieces are described to be at wrong chessboard locations.
Remark: In order to make the AI vision more successful, we increased the size of the chessboard frame tick labels, and turned the “a÷h” ticks uppercase (into “A÷H” ticks.) It is interesting to compare the vision results over chess positions with and without that transformation.
LLM Functions
Let us show more programmatic utilization of the vision capabilities.
Here is the workflow we consider:
Ingest an image file and encode it into a Base64 string
Make an LLM configuration with that image string (and a suitable model)
Synthesize a response to a basic request (like, image description)
Using LLMSynthesize
Make an LLM function for asking different questions over image
Using LLMFunction
Ask questions and verify results
⚠️ Answers to “hard” numerical questions are often wrong.
It might be useful to get formatted outputs
Remark: The function LLMVisionSynthesize combines LLMSynthesize and step 2. The function LLMVisionFunction combines LLMFunction and step 2.
Here we synthesize a response of a image description request:
LLMVisionSynthesize["Describe the image.", imgBarChart,"MaxTokens"->600]
The image shows a bar chart infographic titled "Cyber Week Spending \
Set to Hit New Highs in 2023" with a subtitle "Estimated online \
spending on Thanksgiving weekend in the United States." There are \
bars for five years (2019, 2020, 2021, 2022, and 2023) across three \
significant shopping days: Thanksgiving Day, Black Friday, and Cyber \
Monday.
The bars represent the spending amounts, with different colors for \
each year. The spending for 2019 is shown in navy blue, 2020 in a \
lighter blue, 2021 in yellow, 2022 in darker yellow, and 2023 in dark \
yellow, with a pattern that clearly indicates the 2023 data is a \
forecast.
From the graph, one can observe an increasing trend in estimated \
online spending, with the forecast for 2023 being the highest across \
all three days. The graph also has an icon that represents online \
shopping, consisting of a computer monitor with a shopping tag.
At the bottom of the infographic, there is a note that says the \
data's source is Adobe Analytics. The image also contains the \
Statista logo, which indicates that this graphic might have been \
created or distributed by Statista, a company that specializes in \
market and consumer data. Additionally, there are Creative Commons \
(CC) icons, signifying the sharing and use permissions of the graphic.
It's important to note that without specific numbers, I cannot \
provide actual figures, but the visual trend is clear -- \
there is substantial year-over-year growth in online spending during \
these key shopping dates, culminating in a forecasted peak for 2023.
Repeated questioning
Here we define an LLM function that allows multiple question request invocations over the image:
fst = LLMVisionFunction["For the given image answer the question: ``. Be as concise as possible in your answers.", imgBarChart,"MaxTokens"->300]
fst["How many years are presented in that image?"]
"Five years are presented in the image."
fst["Which year has the highest value? What is that value?"]
"2023 has the highest value, which is approximately $11B on Cyber Monday."
Remark: Numerical value readings over technical plots or charts seem to be often wrong. Often enough, OpenAI’s vision model warns about this in the responses.
Formatted output
Here we make a function with a specially formatted output that can be more easily integrated in (larger) workflows:
fjs = LLMVisionFunction["How many `1` per `2`? " <> LLMPrompt["NothingElse"]["JSON"], imgBarChart,"MaxTokens"->300,"Temperature"->0.1]
Here we invoke that function (in order to get the money per year “seen” by OpenAI’s vision):
Remark: The above result should be structured as shopping-day:year:value. But occasionally it might be structured as year::shopping-day::value. In the latter case just re-run LLM invocation.
Here we parse the obtained JSON into WL association structure:
Remark: Currently LLMVisionFunction does not have an interpreter (or “form”) parameter as LLMFunction does. This can be seen as one of the reasons to include LLMVisionFunction in the “LLMFunctions” framework.
Here we convert the money strings into money quantities:
Here is the corresponding bar chart and the original bar chart (for
comparison):
Remark: The comparison shows “pretty good vision” by OpenAI! But, again, small (or maybe significant) discrepancies are observed.
Dedicated notebook cells
In the context of the “well-established” notebook solutions OpenAIMode, [AAp2], or Chatbook,
[WRIp2], we can contemplate extensions to integrate OpenAI’s vision service.
The main challenges here include determining how users will specify images in the notebook, such as through URLs, file names, or Base64 strings, each with unique considerations. Additionally, we have to explore how best to enable users to input prompts or requests for image processing by the AI/LLM service.
This integration, while valuable, it is not my immediate focus as there are programmatic ways to access OpenAI’s vision service already. (See the previous sections.)
Combinations (fairy tale generation)
Consider the following computational workflow for making fairy tales:
Draw or LLM-generate a few images that characterize parts of a story.
Narrate the images using the LLM “vision” functionality.
Use an LLM to generate a story over the narrations.
Remark: Multi-modal LLM / AI systems already combine steps 2 and 3.
Remark: The workflow above (after it is programmed) can be executed multiple times until satisfactory results are obtained.
Here are image generations using DALL-E for four different requests with the same illustrator name in them:
storyImages =Map[ ImageSynthesize["Painting in the style of John Bauer of " <> #] &,{"a girl gets a basket with wine and food for her grandma.","a big bear meets a girl carrying a basket in the forest.","a girl that gives food from a basket to a big bear.","a big bear builds a new house for girl's grandma."}];storyImages //Length(*4*)
Here we display the images:
storyImages
Here we get the image narrations (via the OpenAI’s “vision service”):
storyImagesDescriptions = LLMVisionSynthesize["Concisely describe the images.", storyImages,"MaxTokens"->600]
1. A painting of a woman in a traditional outfit reaching into a
basket filled with vegetables and bread beside a bottle.
2. An illustration of a person in a cloak holding a bucket and
standing next to a large bear in a forest.
3. An artwork depicting a person sitting naked by a birch tree,
sharing a cake with a small bear.
4. A picture of a person in a folk costume sitting next to a bear
with a ladder leaning against a house.
Here we extract the descriptions into a list:
descr =StringSplit[storyImagesDescriptions,"\n"];
Here we generate the story from the descriptions above (using OpenAI’s ChatGPT):
LLMSynthesize[{"Write a story that fits the following four descriptions:",Sequence @@ descr}, LLMEvaluator -> LLMConfiguration["MaxTokens"->1200]]
In a small village nestled deep within a lush forest, lived a woman \
named Anya. She was gentle and kind-hearted, known for her artistic \
talent and love for nature. Anya had a keen eye for capturing the \
beauty of the world around her through her paintings. Each stroke of \
her brush seemed to hold a piece of her soul, and her art touched the \
hearts of all who laid their eyes upon it.
One sunny day, on the outskirts of the village, Anya set up her easel \
amidst a lively farmers' market. In front of her, she placed a large \
canvas, ready to bring her latest vision to life. With her palette \
filled with vibrant colors, she began painting a woman dressed in a \
traditional outfit, delicately reaching into a woven basket filled to \
the brim with fresh vegetables and warm bread. Beside the basket lay \
an empty bottle, hinting at a joyous feast anticipated for the day.
As Anya skillfully brought her painting to life, a cloak-wrapped \
figure caught her attention. Intrigued, she turned her easel slightly \
to capture the essence of this mysterious wanderer standing beside a \
mighty bear deep within the heart of the forest. In her illustration, \
she depicted the cloaked person, holding a bucket, their gaze met by \
the curious eyes of the regal woodland creature. The bond between \
them was palpable, a silent understanding as they stood together, \
guardians of the ancient woods.
Meanwhile, in a clearing not too far away, Anya discovered a scene \
that touched her deeply. She stumbled upon a person sitting naked \
beneath the shade of a majestic birch tree, a cake placed lovingly \
between them and a small bear. The artwork she created was a tender \
portrayal of the intimate connection shared by the two, a testament \
to the innate kindness that existed between species. Together, they \
enjoyed the sweet treat, their hearts entwined in empathy, neither \
fearing the vulnerability of their exposed selves.
Driven by her artistry, Anya's imagination continued to explore the \
fascinating relationship between humans and bears in her village. In \
her final artwork, she turned her focus to a person in a folk \
costume, sitting comfortably beside a towering bear. A ladder leaned \
against a charming wooden house in the background, illustrating the \
close bond shared between the village folks and their wild \
companions. Together, they stood tall, their spirits entwined in a \
balance of mutual respect and harmony.
As Anya showcased her artwork to the villagers, they were captivated \
by the depth of emotion expressed through her brushstrokes. Her \
paintings served as a reminder that love and understanding knew no \
boundaries, whether lived within the confines of villages or amidst \
the enchanting wilderness.
Anya became a celebrated artist, known far and wide for her ability \
to weave tales of compassion and unity through her exquisite \
paintings. Her work inspired generations to see the world through the \
lens of empathy, teaching them that even in unconventional \
connections between humans and animals, beauty could be found.
And so, her legacy lived on, her art continuing to touch the hearts \
of those who recognized the profound messages hidden within her \
strokes of color. For in every stroke, Anya immortalized the timeless \
bond between humanity and the natural world, forever reminding us of \
the kinship we share with the creatures that roam our earth.
Conclusions and leftover comments
The new OpenAI vision model, “gpt-4-vision-preview”, as all LLMs produces too much words, and it has to be reined in and restricted.
The functions LLMVisionSynthesize and LLMVisionFunction have to be part of the “LLMFunctions” framework.
For example, “LLMVision*” functions do not have an interpreter (or “form”) argument.
The package “LLMVision” is meant to be simple and direct, not covering all angles.
The Nightcore transformation of the video was fairly straightforward with Mathematica / WL. The video transformation and combination are also fairly straightforward or easy.
I downloaded the videos from after searching yandex.ru (dzen.ru). Alternatively, one can find and download videos in Firefox or Google Chrome via relevant plugins. (Or use VLC; or utilize the paclet described in the post “Playing with YouTube from Mathematica”, [BMI1].)
At this point I have a small official video and larger one. This gives the opportunity to demonstrate transferring of the “Dolphin” signature from the “official” video to the larger one. (See the frame manipulation below.)
The process for making a song to be in the Nightcore style is described in Wikipedia, [Wk1]. Basically, we just make the tempo 20-30% faster and raise the pitch with, $\approx 5.5$ semitones.
Remark: An alternative of the process shown in this section is to use audio transformation programs like Audacity and AmadeusPro.
Remark: Here we do not export the video, since Mathematica saves it in a “standard” location of the host operating system.
Cutting-off and re-splicing the movie credits
In order to engage better people from the Millennials and Gen Z generational cohorts, I want to move the movie credits from the start of the movie to be at the end. We use the function VideoSplit, [WRI10], and VideoJoin, [WRI11].
Here we show frames that indicate where to split the obtained Nightcore movie:
Two weeks ago (June 1st and 2nd) I participated in the Wolfram Language conference in St. Petersburg, Russia.
Here are the corresponding announcements:
Interestingly, I first prepared a Latent Semantic Analysis (LSA) talk, but then found out that the organizers listed another talk I discussed with them, on extending dynamic systems models. (The latter one is the first we discussed, so, it was my “fault” that I wanted to talk about LSA.)
Here are the presentation notebooks for LSA in English and Russian .
Some afterthoughts
Тhe conference was very “strong”, my presentation was the “weakest.”
With “strong” I refer to the content and style of the presentations.
This was also the first scientific presentation I gave in Russian. I also got a participation diploma .
I prepared the initial models of artillery shells manufacturing, but much more work has to be done in order to have a meaningful article or presentation. (Hopefully, I am finishing that soon.)
Mathematica for prediction algorithms 
