Remarks from Mark

[amirrr]

• Remove grayed out (disabled) menu items.
• We couldn't remove features or not-include them from a project
• Drag and drop didn't work for papers on a project
• Login page still requires login every time

1. We need to accept comparisons, uploads, and infrastructure for
2. Display ground truth (A way to document it to share our accuracy)
3. Showing quality to user.

[amirrr]

TODOs with the new truth interface:

• Save accuracy by project - (Later to be expanded to platform wide)
• Accuracy should not be bucketed and the raw value determines the color
• Display a meter for the grading (0 red to 1 green)
• Turn the calculation into a long running process
• fix F1 macro/micro to follow formula in observable notebook

[amirrr]

• Try to look for a certain feature in a certain depth, for example the second condition in the first experiment, and they answer 5 questions from that condition. Then try to ask it the same thing on 5 different questions from the same level and compare.
• Run the same paper maybe 10 times to see the variance in the answers it is giving.
• Run the same papers (use a set we benchmark from) on all openai models, compare price, compare accuracy.
• For whatever model best from last step, run ten times. Look at how much we learn failure (1 out 10 disagreement means 90% accuracy) -> are there features that are just 50% (worse than chance)
• Do the first task with categorical features -> they give a better idea if it is doing right or wrong

[amirrr]

For adding new features we want a separate place where users can define and test their new features

experiment name into -> categorical name (how does this new feature do and