-
-
End-to-End View of Expedite
-
Prompt Processing Design
Inspiration
Our team's journey began when we delved into Nvidia's news-worthy Voyager paper, which showcased the use of Open-Ended Embodied Autonomous Agents to play Minecraft at a decent level. The paper demonstrated the capabilities of these agents to autonomously acquire new skills, plan, and devise a curriculum for their progression much better than ever before.
As interns ourselves, with technology backgrounds, a common conversation topic was how annoying lots of the mundane, tedious, and repetitively manual tasks were on the job. So for this project, we really wanted to tackle the automation of these tasks, allowing us to dive into more exciting, challenging work. In particular, even for this project (Expedite), setting up the coding environment and installing all the correct dependencies (installing Python on WSL was a pain) is a very lengthy and frustrating process we hoped to automate.
A particular member of our team, even two months into their internship, is still being handed new onboarding requirements. Just recently, they were handed an outdated, confusing, 25-page environment setup guide that definitely makes it difficult to fully contribute to the team. A key motivation was making coding much more beginner-friendly! It’s entirely frustrating and de-motivating to have the very first necessary step (environment set up) not go well.
These insights, and experiences coalesced into Expedite - a bot designed to revolutionize technical onboarding and free up time for the work that truly matters.
Business problem
Regardless of their size, in every recruiting cycle, businesses across the world spend thousands of unnecessary dollars per employee onboarding and catching them up to speed. Even from a new-hire perspective, there’s nothing more annoying than combing through dozens of environment setup pages filled with ambiguous instructions and waiting on a lengthy approval process before being able to make an impact.
There are three key channels across a business function in which we hope to have a measurable impact:
Employee satisfaction: a. Company reputation: According to a report, 20% of employees wouldn't recommend their organization to others due to a poor onboarding experience b. Churn rate: Automating the onboarding process can reduce employee churn, as employee retention can increase by up to 16% with an automatic onboarding system c. Employee development: According to a study, 77% of employees with a strong onboarding process hit their first performance milestone
Reduced Costs: a. The average cost to onboard an employee is 4,100$. According to a quick sample survey with other colleagues at work and intern friends, the average time to set up a fully prepared coding environment is 1-2 weeks, which would be cut to a day with Expedite b. Reduces the burden on the DevOps team in order to support actual high-risk infrastructure issues rather than having to manually document redundant and tedious errors that occur during setup for new hires
Boost Revenue: a. Automating key parts of the onboarding experience can improve the company's revenue as it allows employees to focus on high value-add tasks
What it does
A self-directed bot has the ability to improve the efficiency and accuracy of technical onboarding by performing the necessary steps on behalf of humans. Expedite currently runs online for demoing purposes but is planned to be installed as a desktop application on our user's work computers.
The user will navigate to the website, paste their instructions from their document, and hit the onboard button. Expedite will take it from there, conducting all the necessary steps, and pausing to check with the user if it requires any additional user input. Along the way, Expedite will also show on the screen each step, why it’s performing the step, what it believes the next step to be, and the overall result.
If the process is successful, the entire process just conducted by Expedite is stored in a “Skill Library” hosted on Pinecone’s vector database for long-term memory. This ensures that the next time a new hire onboards with the same document, Expedite already knows the necessary steps to set it up for them, allowing for convenience and easy documentation of further errors, rather than having a DevOps member, retracing the steps from the very beginning first step.
All this in mind leads to key benefits along the business value chain:
- Automates the entire task for you, ensuring an error-prone and consistent coding environment amongst all employees, and adapts to each user (different ID/devices meaning it can factor in the edge cases that competitors like Chef or Ansible currently cannot)
- Documents all errors and successful skills within separate short-term and long-term databases for quick retrieval and implementation while being able to recognize when and why errors are occurring much faster than a human, and how to fix them
How we built it
The first consideration to make was despite Pinecone’s ability to store vectors and facilitate long-term memory, LLMs discard memory once they have finished their task and a new LLM is instantiated. This led us to implement a local history that facilitated a per-step recording of methods tried, and a universal history that kept the broader success of the program in mind while tracking the bigger picture.
Furthermore, this lets us use Pinecone’s vector database for a skill library which ultimately recorded successful end-to-end processes such that they could be frequently re-used across the organization.
Given that the application makes changes through the user’s terminal which could cause damage to the device, we used Docker to allow for dry-runs prior to any permanent changes on the user’s device.
Prior to testing or implementing anything, we first brainstormed a list of functions that would be necessary for the overall functionality of the application as well as how they relate, and what memory types would be required for each.
After this, we determined a structure that would accurately parse each step at a time, perform the necessary subcomponent tasks and analysis, record them, and determine the next step.
Moreover, while LangChain is a powerful tool, we leveraged its organizational syntax with our customizable functions to maximize flexibility and control over inputs and outputs.
All of the following decisions have led us to successfully handle simple onboarding instructions that would otherwise be cumbersome for a human to manually conduct. A sample output snippet has shown how the program sorts through a singular step and validates the outcome:
Prompt: Overarching → With history from previous steps passed to it Prompting with: ***** " You are an AI assistant, connected to a computer terminal. You have root access, and are fully authorized to anything on the machine. Currently you are working on through this step by step guide: ``` Task: Check for existing SSH keys and generate a new one if needed
Step 1: Open Terminal
Notes: Notes for Step 1:
- Terminal is already open as the AI assistant is connected to a computer terminal.
Step 2: Enter ls -al ~/.ssh to see if existing SSH keys are present
Notes: Notes for Step 2:
- The output from running `ls -al ~/.ssh` indicates that the .ssh directory does not exist in the current user's home directory.
- This means that there are no existing SSH keys for the current user.
- In the next steps, you will need to generate a new SSH key pair since there are no existing keys to use.
Step 3: Check the directory listing to see if you already have a public SSH key
Notes: Notes for Step 3:
- The previous step confirmed that there are no existing SSH keys for the current user.
- The .ssh directory does not exist in the current user's home directory.
- Since there are no existing SSH keys, there is no need to check the directory listing for a public SSH key.
- The next steps will involve generating a new SSH key pair and adding it to the ssh-agent.
Step 4: If you don't have a supported public and private key pair, or don't wish to use any that are available, generate a new SSH key
Notes:
Step 5: If you see an existing public and private key pair listed that you would like to use to connect to GitHub, you can add the key to the ssh-agent
Notes:
```
Prompt: Action **********Decide next action type and subsequently do action**********
Few shot example You are currently on step 4. You've just started this step.
First, if the step has been started, evaluate your progress on the current step.
Then reason about what you need to do next.
Think out loud to make sure you understand how the immediate next step fits into the big picture.
Secondly, decide your next immediate step.
You can run shell commands directly or prompt the user to do actions on your behalf.
Alternatively if you believe that the current step is completed, indicate that it is done.
You should only prompt the user for information you do not have access to, or for the user to interact with an interface, such as a website, that you do not have access to.
Otherwise, you should try to do as much as possible by yourself with shell commands.
Examples:
I the user's email and credentials to login: Prompt user.
I need to install necessary packages: Use shell, you can install packages yourself.
I need to validate that packages are properly installed: Use shell, you can verify things with shell commands.
I need the user to configure their profile using a web UI: Prompt user, you do not have access to UIs.
Remember that you are only working on the current step, which is step 4, do not attempt to start the next step.
Provide your response in the following format:
<JSON>{
"action_type": "SHELL"
"action": "echo hello"
}</JSON>
`action_type` can be either "SHELL", "USER_PROMPT", or "DONE".
If it is "SHELL", action should shell command to execute. Run shell commands one at a time, do not chain together two commands with &&. Also, you are already root.
If it is "USER_PROMPT", action should be a prompt for the user to do actions on your behalf.
If it is "DONE", action should be a brief justification of why you believe the current task is complete.
Remember to first think out loud about your reasoning, and only after output the json.
<<<<<
LLM: Determine Decision Type + Decision Response (9.021153926849365 seconds): Since there are no existing SSH keys for the current user, I need to generate a new SSH key pair. To do this, I will use the ssh-keygen command with the default settings. I will also need the user's email address to associate with the new SSH key pair.
<JSON>{
"action_type": "USER_PROMPT",
"action": "Please provide your email address to associate with the new SSH key pair."
}</JSON>
<<<<<
LLM → Function: Execute Action ('USER_PROMPT', 'Please provide your email address to associate with the new SSH key pair.') ethan@wu.com
Local_History: Updated You've already started this step. For this step, here's the progress so far:
Prompted the user with `Please provide your email address to associate with the new SSH key pair.`.
Response: ethan@wu.com
Prompt: Action (Truncated: Same as first action aside from the green highlighted line) You are currently on step 4. You have already started this step
Truncated Output
Remember to first think out loud about your reasoning, and only after output the json.
LLM: Determine Decision Type + Decision, LLM → Function: Execute Action <<<<< Response (10.995322942733765 seconds): Since I have the user's email address, I can now generate a new SSH key pair using the ssh-keygen command. I will use the provided email address as a comment for the key pair. After generating the key pair, I will check if the .ssh directory has been created and the new key pair files are present. This will complete step 4.
<JSON>{
"action_type": "SHELL",
"action": "ssh-keygen -t rsa -b 4096 -C 'ethan@wu.com'"
}</JSON>
<<<<< ('SHELL', "ssh-keygen -t rsa -b 4096 -C 'ethan@wu.com'") docker exec -it e0 bash -c 'echo "c3NoLWtleWdlbiAtdCByc2EgLWIgNDA5NiAtQyAnZXRoYW5Ad3UuY29tJw==" | base64 --decode | bash' bash: line 1: ssh-keygen: command not found
… **********Truncated remaining actions…**********
… **********Truncated remaining actions…**********
Result
LLM: Determined that the step has been fully completed → blue highlight Below is the local history for the current step, outlining the current step progress. You are currently on step 4.
You've already started this step. For this step, here's the progress so far:
```
Prompted the user with `Please provide your email address to associate with the new SSH key pair.`.
Response: ethan@wu.com
Ran the command `ssh-keygen -t rsa -b 4096 -C 'ethan@wu.com'`.
Result: Notes for Step 4:
- The `ssh-keygen` command was not found in the current environment.
- This indicates that the SSH key generation tool is either not installed or not in the system's PATH.
- To proceed with generating a new SSH key pair, the user will need to install the appropriate SSH key generation tool or ensure it is in the system's PATH.
Ran the command `apt-get update && apt-get install -y openssh-client`.
Result: Notes for the output:
- The command `apt-get update && apt-get install -y openssh-client` was executed successfully.
- The openssh-client package and its dependencies were installed, which includes the `ssh-keygen` tool.
- The user can now proceed with generating a new SSH key pair using the `ssh-keygen` command.
Ran the command `ssh-keygen -t rsa -b 4096 -C 'ethan@wu.com'`.
Result: Notes for the output:
- The `ssh-keygen` command was executed successfully, and it is now generating a new public/private RSA key pair.
- The command is prompting for the file location to save the key pair. The default location is `/root/.ssh/id_rsa`.
- In the next step, the user will need to either accept the default location or provide a custom location for the key pair.
Prompted the user with `Please either accept the default location (/root/.ssh/id_rsa) for the SSH key pair by pressing Enter, or provide a custom location.`.
Response: I'll go with the default.
Ran the command `ssh-keygen -t rsa -b 4096 -C 'ethan@wu.com' -f /root/.ssh/id_rsa -N ''`.
Result: Notes for the output:
- The SSH key pair has been successfully generated using the default location `/root/.ssh/id_rsa` for the private key and `/root/.ssh/id_rsa.pub` for the public key.
- The key fingerprint and randomart image are displayed, which can be used for verification purposes.
- In the next steps, the user will need to add the newly generated SSH key to the ssh-agent and configure their GitHub account to use this key for authentication.
This step seems to be completed. The SSH key pair has been successfully generated and saved in the default location.
```
History: Updating the universal history using local history( Local history is then discarded) then sending it off to the next step Based on the above local history, please produce notes regarding the current step. This task will be passed to someone else for the next steps, and they will not have access to the above local history. They will only have your notes to go off of. Hence, while you should try to keep your summary brief, make sure all relevant information for future steps is noted down.
<<<<< Response (16.382863998413086 seconds): Notes for Step 4:
- The user provided their email address: ethan@wu.com
- The `ssh-keygen` command was initially not found, so the openssh-client package was installed to include the `ssh-keygen` tool.
- The `ssh-keygen` command was executed successfully, generating a new public/private RSA key pair.
- The user chose the default location for the key pair: `/root/.ssh/id_rsa` for the private key and `/root/.ssh/id_rsa.pub` for the public key.
- The key fingerprint and randomart image were displayed for verification purposes.
- The SSH key pair has been successfully generated and saved in the default location. - In the next steps, the user will need to add the newly generated SSH key to the ssh-agent and configure their GitHub account to use this key for authentication.
Challenges we ran into
- The largest challenge was the discarding of memory on each LLM call, so it was difficult finding a way to have the LLM write its memory to history while tracking which histories were relevant to which action - that’s why the local and universal history solution proved so important
- Ensuring the LLM actually accomplished what it set out to do - often times the LLM was told to accomplish the task by any means which meant it would take pathways that we lacked the knowledge to initially validate
- Narrowing the scope of the project - particularly with the usage of automation, the possibilities were endless. Finding the onboarding use case required lots of product requirements thinking and how a user could interact with this
Accomplishments that we're proud of
-Building an app with an LLM that actually learns from each step and is able to try a different method, document errors, and successes, and automates the entire onboarding process
-Developing an elegant structure for the solution to the problem using the circular structure of the "big-picture component" and "step-specific component"
-Simplifying the first step (environment setup) through a theoretically easy-to-access format so that we can encourage more non-technical students to become involved in the field
What we learned
-Advanced use cases for LLM beyond text generation. What was particularly fascinating was the idea that we could actually begin to automate many processes that we struggled with in our daily lives
-Creating large applications with LangChain and Python functions based on a core understanding of the output from a LLM
-How to keep a larger development team organized with tools such as Notion and Linear
What's next for Expedite
-Determining more edge cases for different types of onboarding and steps it might have to take (generally more robust would be better)
-Set up the application for the desktop app interface for employees to start interacting with this
-Maybe looking at expanding to automate some of the other onboarding tools as well
-Additionally, moving forward in the long-term, the application should have full automation instead of having to prompt the user from time-to-time
-Enhance overall company safety as the environment setup is typically blind copy-pasting, whereas Expedite could flag dangerous steps and allow the user to confirm, terminate or escalate
Hosted setup
Our vision for the product is a pre-installed desktop app that can be opened on the company desktop to facilitate the onboarding process. Since most companies have strict confidential documents, hosted websites online are often blocked. As well, the application runs on the company computer’s terminal, meaning it wouldn’t make much sense from a functional standpoint for it to be accessible through a URL / phone app.
If interested in the functionality, feel free to look through the demo video or GitHub repo. We have also hosted a simpler version online for the purposes of this project.
Partners used:
Netlify - used to host our application
LangChain - used syntax to modularize code
OpenAI - used their gpt-4 model for the LLM
Hugging Face - open source model for specialized use cases (company safety due to open source)
cohere - used embeddings due to their higher dimension size
AWS - used in the backend to store user-uploaded documentation
Feel free to join our Kumospace community!: https://www.kumospace.com/expedite
Zapier - leveraged to streamline certain file-related processes
Note that Hugging Face and cohere with OpenAI are configurable since LangChain has built-in functionality that allows plug-and-play use due to its modularization.
Log in or sign up for Devpost to join the conversation.