fix!: merge payload and data fields of Request

[vdusek]

Description

• We had data and payload fields on the Request model.
  • payload was not being provided to the HTTP clients, only the data field.
• In this PR, I'm merging them together, keeping only the data field (use the same naming as HTTPX & CurlImpersonate).
• In this PR, I'm merging them together, keeping only the payload field (use the same naming as in JS Crawlee).
• I also defined type aliases for HTTP data and HTTP query parameters and used them across the project.
• Some struggle with Pydantic & serialization, but should be OK now...

Issues

• N/A (ad-hoc fix; reported by @honzajavorek on Slack)

Testing

• The current set of unit tests should cover the changes.

Checklist

• CI passed

[B4nan]

JS version have only payload, we should aim for compact with that in the first place.

[B4nan]

Also, since this is breaking, maybe we should wait, and merge it once we have more breaking changes than a simple one like this (or some feature worth the minor bump).

[honzajavorek]

I learned about data here, but I don't see it in changed files: https://crawlee.dev/python/docs/examples/fill-and-submit-web-form#preparing-a-post-request

[honzajavorek]

It also says "Alternatively, you can send form data as URL parameters using the query_params argument." btw, not sure if that is something related or not.

[vdusek]

Also, since this is breaking, maybe we should wait, and merge it once we have more breaking changes than a simple one like this (or some feature worth the minor bump).

We will have the first version of fingerprints for Playwright soon 🤞.

I learned about data here, but I don't see it in changed files: https://crawlee.dev/python/docs/examples/fill-and-submit-web-form#preparing-a-post-request

Thanks. Of course, I forgot to update the docs.

[janbuchar]

Nice! Just a couple of questions...

[janbuchar]

The dict is kinda surprising here - I'd expect bytes or something. How did this happen? 😄

[vdusek]

The downstream libraries (HTTPX & CurlImpersonate) do the conversions. And both of them accept RequestData = Mapping[str, Any] type.

[janbuchar]

Right. I did some digging and this is what I found. payload is part of the Request schema on the API, and the API client requires it to be a string. It'll get serialized and passed to the API endpoint on platform. Of course, it also needs to be compatible with whatever the HTTP client accepts - that's where the actual processing gets done.

As a side note, I'm not sure what data was supposed to be, but it seems it's not needed...

[vdusek]

Also, since this is breaking, maybe we should wait, and merge it once we have more breaking changes than a simple one like this (or some feature worth the minor bump).

I've separated the Curl impersonation fix and other minor updates into a separate PR (#543), allowing us to release a patch version for 0.3.

The payload & data merge and HttpHeaders thing will be resolved later...