BRIGHTCODE – Michał Żarnecki Portfolio

Implemented a collection of tools that represent different strategies for evaluating LLM responses in most popular PHP AI/LLM framework LLPhant.
Supported 10 different strategies for evaluating LLM responses:

Score evaluators:

• Criteria evaluator
• Embedding distance evaluator
• String comparison evaluator
• Trajectory evaluator

Output validation:

• JSON format validator
• XML format validator
• Fallback messages validator
• Regex pattern validator
• Token limit validator
• Word limit validator

Introduced A/B testing for different LLMs response comparison.
Added also guardrails which are lightweight, programmable checkpoints that sit between application and the LLM. After each model response they run an evaluator of your choice (e.g. JSON‐syntax checker, “no fallback” detector). Based on the result, either pass the answer through, retry the call, block it, or route it to a custom callback.

Framework repository: https://github.com/LLPhant/LLPhant
Evaluation module: https://github.com/LLPhant/LLPhant/tree/main/src/Evaluation

In this project I built a pipeline that transforms raw corporate‑registry filings into an interactive ownership‑tree dossier. This approach helps analyse even complex company structures and detect beneficial owners in multi‑level hierarchies. The application reconstructs every legal entity in a group, calculates the equity stake at each level, and flags individual shareholders who meet Germany’s ≥ 25 % “wirtschaftlich Berechtigter” threshold. A single PDF (example attached) contains the full tree, drill‑down tables, and a concise write‑up in both German and English.

Functional Overview

• Document acquisition – Scrapes excerpts, shareholder lists and capital‑table attachments; deduplicates, corrects, matches and standardises addresses, register numbers and legal forms.
• Graph‑builder – A Neo4j‑backed engine converts the filings into a directed equity graph. Direct and indirect holdings are aggregated with depth‑first traversal to produce the true economic ownership percentages.
• LLM reasoning chain – Specialised prompts guide LLM to parse unstructured shareholder related documents, providing structured shareholder information
• Visual presentation – D3 js library renders tree also handling cricular dependencies and output is captured by pupeteer service in order to generate final PDF with clickable deep‑links back to source documents.

In this project I designed pipeline that converts raw financial statements and company data into a fully formatted credit-rating dossier. The application produces a narrative assessment, multi-year ratio analysis, yer-to-year comparison and a suite of visualisations each generated dynamically at runtime.

Functional Overview

• Data acquisition – The service ingests three years of balance-sheet, income-statement, and cash-flow data together with current market quotations.
• Analytical engine – A carefully curated Expert-Investor system prompt guides OpenAI o3 and Claude 3.7 sonet to draft commentary in German, embedding all referenced figures verbatim for improved traceability. Guard-rails prevent from wrong output formatting, validate and correct data if any issue is detected. Analysis consists multiple stages where output of single LLM with secialized prompt is passed to another LLM with different instructions and by such multi stage reasoning, complete, repeatable and high quality of expertise is achived.
• Credit scoring – Liquidity, solvency, and profitability ratios are computed and weighted to deliver an internal credit rating.
• Visual presentation – Matplotlib/Seaborn charts (trend lines, ratio heat maps) are embedded directly into the final PDF.

Business Value

Automated, audit-ready financial commentary shortens the traditional analyst workflow and enables decision-makers without a finance background to evaluate corporate health immediately.

>>> see sample report <<<

For more technical details and guidance how to approach such topic please refer also to my article “Building Financial Reports with FinGPT and Claude 3.7 Sonnet” (Medium, May 2025).

ConFoo 2025, held February 26-28 in Montreal, Canada. ConFoo.ca is a premier multi-technology conference for developers that brings together international speakers focused on delivering pragmatic solutions for today’s development challenges .

At the conference, I presented two technical sessions that highlighted expertise in artificial intelligence and natural language processing applications:

First presentation, “Retrieval Augmented Generation in PHP,” shared our experience and methodologies for implementing RAG architecture to extract precise information from unstructured data and evaluate output in measurable way.

Second session, “Replacing Complex Business Logic with Machine Learning,” demonstrated how our team has implemented ML and GenAI solutions to streamline previously difficult-to-maintain systems. The session featured case studies from actual projects, illustrating the tangible benefits of transitioning from rule-based systems to machine learning approaches, especially in parsing documents and handling large volumes of unstructured text data.

During my talk I shared the journey of building a system that maps unstructured company descriptions to official industry codes which is challenging because of 1800 possible classification categories and multiclass classification.

In presentation I described how over the years, we’ve evolved our solutions:
– From random forest classifier
– To zero-shot + classification dedicated Large Language Model (LLM)
– And finally, LLMs + Retrieval Augmented Generation (RAG)

System for identification of similar companies and competitors based on company description text, texts from company websites, coordinates, industry tags, revenue and size of employment.
System is based on relational database queries that include logistic regression model coefficient trained on large dataset related to German companies.

Solution for multi-class classification of unstructured texts containing company descriptions categorized into over 1800 classes (WZ 2008), using Python libraries, large language models (LLMs) and the Retrieval Augmented Generation (RAG) technique.
This project was evolving since 2021 together with artificial inteligence solutions and initially approached using a Random Forest Classifier, to be later replaced with zero-shot transformer based solution and finally solved with a solution based on RAG and LLMs, yielding significantly improved results.
More details can be found in article

Lecture: Classifying unstructured texts into 1800 categories!
Problem: In this presentation, I will examine the development of a text classifier created by the team at CompanyHouse AG to address the challenge of classifying unstructured texts that describe companies’ activities into the official German industry codes, WZ 2008. Over the years, we have experimented with various techniques to manage classification across a vast number of categories (1,800 in total). I will discuss the strategies we employed to tackle this complexity and demonstrate the evolution of our model from a random forest classifier to an innovative solution based on large language models and retrieval-augmented generation (RAG) techniques.

Methodology: Our approach includes a range of methodologies: multiclass classification, retrieval-augmented generation (RAG), random forest classifiers, similarity algorithms, embedding techniques, and the use of vector databases.

Conclusions: Integrating additional knowledge into models using retrieval-augmented generation combined with similarity algorithms and techniques such as chain-of-thought reasoning can effectively address complex multiclass classification problems. This approach achieves high evaluation scores and outperforms pre-trained classifiers.

The course familiarizes participants with aspects of generative artificial intelligence and the latest achievements in the field of natural language processing. Participants learn the theoretical foundations and architecture of Large Language Models (LLM) and gain practical skills in working with text data. The course places particular emphasis on understanding popular tasks using LLM such as text generation, machine translation, sentiment analysis, creating summaries and answering questions based on a database of source documents. After completing the course, the participant knows the techniques used in command engineering, metrics for evaluating the results generated by LLM, and methods for improving returned content. He can also apply large text models to a variety of applications in research, industry and other areas.

topics:
Generative AI tasks: translation, question-answer, summarize, sentiment
Architecture and types of LLM, encoder, decoder.
Text vectorization, positional coding, attention mechanism (Multi-Head Attention)
OpenAI GPT, Google Gemini, Mistral Mixtral, Meta Llama, Claude, FLAN models.
Prompt engineering, multi-task instruction fine-tuning, zero/one/few shot learning
Parameter-Efficient Fine-Tuning (PEFT), LoRA
Division of instructions into steps: chain-of-thought
Evaluation of LLM models: performance evaluation, ROUGE/BLEU metrics, benchmark
RAG – Retrieval Augmented Generation, vector database
LLM training computational challenges, scaling laws
Frameworks for working with LLM, LangChain, ReAct