Project Penelope
{ INTRO }
PROJECT OVERVIEW
CONTEXT
Having witnessed the sudden appearance of over dozen new AI models, I took an interest in creating my own.
This decision drove me right into the massive brick wall which is requiring training data.
There was simply no way I could harvest enough data needed to create my very own model, not to mention the processing power I would also need.
Having reached this inevitable barrier, I decided if I couldn't make one then I would simply need to modify an already existing one to suit my needs.
RUNDOWN
Project Penelope was developed using an abundance of different software, APIs and programming languages, with Python being at the forefront of the backend.
Penelope, for lack of a better term, is an incredibly advanced chatbot.
It can recieve input through text or verbal speech and output it's own vocalized response dependant on a few factors.
These factors are: personality, emotion, context and long-term memory.
They will determine the manner and tone which Penelope responds, gradually changing as the conversation continues.
Aside from being able to speak, Penelope also has a visual model in a virtual space, allowing for the speaker to see Penelope's reactions being animated in real-time. The idea for this derived from virtual youtubers (VTubers).
The aforementioned virtual space is inherently it's own independent project, featuring interactive UI elements such as resizable windows, captions, charts, toggles and random effects.
{ FEATURES }
Emotional Aptitude
Penelope is capable of expressing four emotions, excitement, rage, disgust and anxiety.
These emotions gradually shift and overlap during conversations to create a more immersive experience.Character Integrity
Before a response can be generated, two sets of data containing information regarding Penelope's characteristics, built-in memories and perspectives are sifted through in order to establish the most fitting reply. This avoids bland, informative or inexpressive outputs.
Sentiment Analysis
All input received from the end-user is analyzed to break down the sentiment being expressed. This allows Penelope to understand the tone of what's being said, and decide whether it's kind, rude, comedic or simply monotone. This feature also influences Penelope's change in emotion.
DISCORD INTEGRATION
(Hover to reveal)
DISCORD INTEGRATION
An alternative way to communicate directly with Penelope.
STATIC PREFERENCES
(Hover to reveal)
STATIC PREFERENCES
Penelope has preferences in topics such as movies, games, music, sports and more.
These remain consistent regardless of how many times the same topic is brought up, allowing for a sense of identity.
CONTINUOUS RESPONSE
(Hover to reveal)
CONTINUOUS RESPONSE
Penelope doesn't always wait for a user reply.
This creates even greater immersion, since real conversations don't function like a turn-based game.
PENELOPE'S VISUAL SPACE
This is where the magic happens.
Through the use of basic networking techniques, I was able to create a communication link between the visual frontend and the intricate backend, despite both being developed through different tools, software and programming languages. (Python/C#)
When the backend that is Penelope wants to say something, it will first generate an audio file containing speech. This audio file is then sent to the visual space which contains it's own unique codebase.
From here the speech audio is played adjacent to a talking animation. Captions can also be optionally displayed in real-time.
The visual space was designed to resemble a miniature desktop screen.
As such, user interface elements take the form of app icons that open up a window when clicked on. This makes the visual space feel welcoming and familiar to any user.
It also makes things very intriguing, allowing Penelope to appear as though it has it's very own playground.
Features worth mentioning:
DYNAMIC WINDOWS
All windows can be dragged, resized, closed and even have their color changed around.
RANDOM EVENTS
The visual space is unpredictable. It may rain, twist, distort or even split in half at random intervals.
DATE & TIME
Accurate date and time is provided while the space is actively running.
{ EXTRA }
MyDEAR
Wordplay for "my disgust, excitement, axiety and rage".
Provides a unique way to visualize Penelope's emotional state changing in real-time.
The design was inspired by the ECG monitors found in hospitals. If a certain one go too high, you know you're in trouble.
Inner Workings:
The backend of Penelope delivers a message to the visual space, notifying it of any changes in emotion.
The respective lines are then animated to reflect this by expanding or shrinking.
The frequency, speed and shakiness of each line can also be manipulated to better express these changes.
RANDOMNESS
In order to liven things up, a variety of random events can occur.
Some of these events directly impact Penelope, altering their voice, image or erasing them from view entirely.
DRAW MODE
Another engaging feature is the draw mode.
As the name suggests, it allows you to doodle anywhere inside the visual space by pressing the F1 key.
Doing so also alerts Penelope, allowing them to comment on your "masterpiece".
GLITCHES & STUTTERS
The personality of Penelope was inspired by Wreck-It Ralph character Vanellope, a mischievous girl inside an arcade game who became a glitch due to her corrupt data.
As a tribute to one of my favorite movies, Penelope inherited this character's glitch.
In practice, whenever Penelope is speaking, there is a very small chance they will appear distorted, repeating the same vowel in quick succession for 1-2 seconds.
There is also a small chance for Penelope to randomly teleport around the visual space.
{ VIDEOS }
<> STARTUP
ANIMATION
<> COMICAL CLIP
<> HORROR
CLIP
{ OVERVIEW }
Tech Stack
- Python
- Unity
- Git
- Blender
- Photoshop
- Microsoft Azure
Python Libraries
- Transformers
- Torch
- Whisper
- PyDub
- Azure
- Asyncio
- Wave
- Pycord
- Socket
- Other
DEVELOPMENT CHALLENGES
Subtle Changes In Emotion
Designing a system where four emotions could blend and overlap rather than randomly switching required careful weighting and priority logic. Penelope needed to express mixed emotional states in a way that felt natural to the user, making emotional transitions and visual feedback significantly more complex.
Network Interoperability
Since the backend logic ran separately from the visual application, all animation cues, captions, and audio triggers had to be synchronized through network messaging. Minimizing latency and handling delayed packets was critical to prevent desynchronization.
Accurate Speech Transcribing
Accurately detecting when the user had finished speaking was challenging, especially with natural pauses in conversation. The system had to balance responsiveness with accuracy to avoid cutting off speech early or introducing noticeable delays before a response.