Thread to discuss our community to do list task of creating a standardized list of evaluation metrics for Digital Brain Evaluations.
Feel free to post any and all ideas for things we should be evaluating digital brain providers on below.
Here are my general thoughts on the minimum requirements for my digital brain:
- Ownership & Control
- Secure
- Smart
- Personalized
- Easy to use
- Mobile first
- Fast
- Voice & Vision
- Modular
Here are more details on each of my requirements:
Requirement 1: Digital Brain Ownership and Control
(Main Image)
Corporate owned digital brains like ChatGPT offer great solutions to all requirements except 1:
Ownership and control.
If we use a corporate owned digital brain like ChatGPT, we don’t own my digital brain, they do.
So that’s why ownership and control should be our #1 requirement for our digital brains.
Requirement 2: Secure
We’ll want to be able to feed all our personal data, knowledge, and information to our digital brains just like we do our physical brains.
So they have to be highly secure.
Requirement 3: Smart
The primary reason I want my own digital brain is because I want it to be an expert in my pocket on any topic.
Siri is a dumb example of a digital brain. I rarely use it.
ChatGPT is a smart example of a digital brain. I use it all the time.
So our personal digital brain needs to be smart.
And when it has access to our personal data, it needs to be smarter than ChatGPT.
If it’s smart we’ll use it.
And if it’s dumb we will not.
Requirement 4: Personalized
The more data we give our digital brain the more personalized its answers, advice, and help should be.
It should remember what we tell it.
And then incorporate our instructions into future conversations without us having to tell it again.
Requirement 5: Easy to use
If it’s not easy to use we’ll just default back to using chatGPT, which is easy to use.
Everything should happen via chat.
It should require no programming.
And no movement away from the chat window.
Requirement 6: Mobile First
I want my digital brain with me at all times.
That means it needs to live on our mobile devices which right now means our Apple and Android smartphones.
It should also work on desktop.
But if there are tradeoffs that need to be made they should be on desktop and not mobile.
Requirement 7: Fast
30 seconds may not sound like much.
But think about having a conversation with a person where there was a 30 second pause between each person talking.
Talking with our digital brain should be at least as fast as talking to a person.
Requirement 8: Voice & Vision
We want to be able to talk to our digital brain as well as type to it.
We also want to be able to show it things.
We’d rather have a human expert that I can talk, text, and show things.
And the same is true of a digital expert.
Requirement 9: Modular
The “state of the art” is changing rapidly in AI.
And we want to be able to use the best components available for our digital brain.
This means it needs to be easy to swap out old components of our digital brain and swap in new components.
When we swap one component out it should work with the other components.
This is what is referred to as a “modular” system.
Have other ideas or think we should change any of this?
Post your thoughts below!