Refactor DSPy adapters to make it more extensible

[chenmoneygithub]

We are reworking DSPy adapters for extensibility. For most users this change shouldn't cause backward compatibility issues, but if your workflow explicitly calls child methods of DSPy adapters, you need to make adjustments.

The goal here is with DSPy 3.0, we want Adapter to be a customizable interface rather than some tribal knowledge. We acknowledge that it will be common for users willing to write their own adapter to adjust to their LLMs and workflows. However, the current adapter doesn't have a decent abstraction, and to write a custom adapter users need to understand the source code, and go through a tedious debugging process without guidelines.

In this PR, we are trying to standardize the dspy Adapters, and open a few hooks for people to override during customization. We are aware that there is no single standard that fits all use cases, but trying to hit a stage where we don't over-simplify or over-engineer the base DSPy Adapter/

In a nutshell, we are making the following breakdown of Adapters:

• Adapter
  • format(): formats the type-based inputs into LM multiturn messages
    • System messages: The high level description of the task, and LM I/O format.
      • Fields description: format_field_description()
      • LM input/output structure description: format_field_structure()
      • task description: format_task_description
    • Few-shot examples (demo): multiturn few-shot examples
      • user message (inputs of demo): format_user_message_content()
      • assistant message (outputs of demo): format_assistant_message_content()
    • Conversation history: multiturn conversation history
      • user message (inputs of history message): format_user_message_content()
      • assistant message (outputs of history message): format_assistant_message_content()
    • Current input: the actual question/input
      • user message: format_user_message_content()
  • parse(): parse the LM response to type-based outputs. No sub-hook for parse() because it varies for different adapters.

Note that format_user_message_content() and format_assistant_message_content() are used in multiple places. Users can override any level of hooks for customization.

We will publish a guide on how to customize Adapter with concrete use cases after landing this PR.