Asynchronous Response support for Data Prepper Admin Apis

[dinujoh]

Is your feature request related to a problem? Please describe.
In containerized environments, direct communication between control plane and data plane (Data Prepper) is restricted. For example when running data prepper as container in ECS Fargate, the control plane can initiate requests (curl commands) using ecs execute-command to Data Prepper's admin APIs, it faces an inherent limitation:

• Can execute commands in the container
• Cannot capture or return the API response
• Command execution succeeds but response is lost

Describe the solution you'd like
Asynchronous responses solve this by inverting the communication flow:

1. Control plane initiates request with callback URL
2. Data Prepper receives request
3. Data Prepper independently calls callback URL with results
4. No need to capture response through execute-command

Here's a high-level overview of supporting both synchronous and asynchronous responses in Data Prepper's admin API:
Request Pattern:

1. Sync Request:
   GET /pipelines
   Response: Immediate result

2. Async Request:
   GET /pipelines
   Header: X-Callback-Url: http://callback-endpoint
   Response: Immediate acknowledgment + Callback with result

Implementation Strategy:

1. Handler Detection:

• Check for X-Callback-Url header or query parameter
• Choose sync/async processing based on presence
• Same endpoint supports both patterns

2. Response Flow:

• Sync: Request -> Process -> Return Result
• Async: Request -> Return Ack -> Process -> Callback with Result

3. Response Types:

• Sync Response: { data: actual_result }
• Async Response: { requestId: "uuid", status: "ACCEPTED" }
• Callback: { requestId: "uuid", data: actual_result }

This approach maintains backward compatibility while adding async support through optional callback URLs.

Describe alternatives you've considered (Optional)
Run a control plane proxy along with Data Prepper that has direct access to data prepper container