TypeError: HTTPRequest fails when processing input text records that need tainting and their position is after file type records #1095

xispa · 2023-01-18T10:39:18Z

BUG/PROBLEM REPORT / FEATURE REQUEST

A TypeError: can't pickle cStringIO.StringO objects is raised when ZPublisher processes a request input record with a value that contains a character that needs to be tainted (e.g. "<") and when a record from "file" type has been processed previously.

When a value from a record should be tainted, ZPublisher does it, but it also does a deepcopyof the records processed earlier here:

Zope/src/ZPublisher/HTTPRequest.py

Lines 725 to 726 in 53d8f7e

    
           tainted_mapping[tainted_key] = deepcopy( 
        
               reclist)

. Thus, if one (or more) of these records processed earlier is from file type, deepcopy cannot do the job and the following traceback arises:

Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 162, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 371, in publish_module
  Module ZPublisher.WSGIPublisher, line 232, in publish
  Module ZPublisher.HTTPRequest, line 737, in processInputs
  Module copy, line 163, in deepcopy
  Module copy, line 230, in _deepcopy_list
  Module copy, line 190, in deepcopy
  Module copy, line 334, in _reconstruct
  Module copy, line 163, in deepcopy
  Module copy, line 257, in _deepcopy_dict
  Module copy, line 190, in deepcopy
  Module copy, line 334, in _reconstruct
  Module copy, line 163, in deepcopy
  Module copy, line 257, in _deepcopy_dict
  Module copy, line 182, in deepcopy
TypeError: can't pickle cStringIO.StringO objects

As a result, the base error screen from plone "We’re sorry, but there seems to be an error…" is rendered, but without any error or traceback information.

What I did:

Tried to submit a form made of the following:

<form name="set-analysis-conditions-form"
                class="form"
                method="POST"
                enctype="multipart/form-data"
                tal:attributes="action string:${here/absolute_url}/set_analysis_conditions">

            <table class="table table-sm small">
              <colgroup>
                <col style="width:20%"/>
              </colgroup>
              <tal:condition repeat="condition python:view.get_conditions()">
                <tr tal:define="required python:condition.get('required') == 'on';
                                is_checkbox python:condition.get('type') == 'checkbox';
                                required python:required and not is_checkbox;">
                  <td>
                    <label class="formQuestion">
                      <span tal:content="structure condition/title"/>
                      <span class="fieldRequired"
                            tal:condition="required"
                            i18n:attributes="title title_required;"
                            title="Required">&nbsp;</span>
                      <span class="formHelp discreet help-block-small"
                            tal:content="structure condition/description|nothing"/>
                    </label>
                  </td>
                  <td tal:define="options condition/options|nothing">

                    <!-- Hidden fields to keep original conditions info -->
                    <input type="hidden" name="conditions.title:records"
                           tal:attributes="value condition/title"/>
                    <input type="hidden" name="conditions.description:records"
                           tal:attributes="value condition/description"/>
                    <input type="hidden" name="conditions.choices:records"
                           tal:attributes="value condition/choices"/>
                    <input type="hidden" name="conditions.uid:records"
                           tal:attributes="value condition/uid"/>
                    <input type="hidden" name="conditions.type:records"
                           tal:attributes="value condition/type"/>
                    <input type="hidden" name="conditions.required:records"
                           tal:attributes="value condition/required"/>
                    <input type="hidden" name="conditions.attachment:records"
                           tal:attributes="value condition/attachment/uid|nothing"/>

                    <select tal:condition="options"
                            tal:define="value condition/value|nothing"
                            name="conditions.value:records">
                      <option tal:condition="not:required"/>
                      <tal:option repeat="option options">
                        <option tal:content="option"
                                tal:condition="python:option == value"
                                selected="selected"/>
                        <option tal:content="option"
                                tal:condition="python:option != value" />
                      </tal:option>
                    </select>

                    <tal:non_select condition="not:options">
                      <input tal:attributes="type condition/type;
                                             value condition/value"
                             tal:condition="required"
                             required="required"
                             name="conditions.value:records"/>

                      <input tal:attributes="type condition/type;
                                             value condition/value"
                             tal:condition="not:required"
                             name="conditions.value:records"/>
                    </tal:non_select>

                    <div tal:define="attachment condition/attachment|nothing"
                         tal:condition="attachment" class="mt-2">

                      <a tal:content="attachment/filename"
                         tal:attributes="href attachment/download_url"/>
                    </div>

                  </td>
                </tr>
              </tal:condition>
            </table>


            <div class="form-group mt-2">
              <input class="btn btn-sm btn-primary"
                     type="submit"
                     name="set_conditions"
                     i18n:attributes="value"
                     value="Set Conditions" />
            </div>

            <!-- hidden fields -->
            <input type="hidden" name="submitted" value="1" />
            <input type="hidden" name="uid"
                   tal:attributes="value python:view.get_uid_from_request()" />
            <input tal:replace="structure context/@@authenticator/authenticator"/>

          </form>

What I expect to happen:

No traceback, records are processed correctly

What actually happened:

Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 162, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 371, in publish_module
  Module ZPublisher.WSGIPublisher, line 232, in publish
  Module ZPublisher.HTTPRequest, line 737, in processInputs
  Module copy, line 163, in deepcopy
  Module copy, line 230, in _deepcopy_list
  Module copy, line 190, in deepcopy
  Module copy, line 334, in _reconstruct
  Module copy, line 163, in deepcopy
  Module copy, line 257, in _deepcopy_dict
  Module copy, line 190, in deepcopy
  Module copy, line 334, in _reconstruct
  Module copy, line 163, in deepcopy
  Module copy, line 257, in _deepcopy_dict
  Module copy, line 182, in deepcopy
TypeError: can't pickle cStringIO.StringO objects

PDB session

TypeError: can't pickle cStringIO.StringO objects
(Pdb++) where
[...]
-> self.execute()
[6]   /home/senaite/dev/buildout-cache/eggs/waitress-1.4.4-py2.7.egg/waitress/task.py(441)execute()
-> app_iter = self.channel.server.application(environ, start_response)
[7]   /home/senaite/dev/buildout-cache/eggs/Paste-3.5.0-py2.7.egg/paste/translogger.py(69)__call__()
-> return self.application(environ, replacement_start_response)
[8]   /home/senaite/dev/buildout-cache/eggs/Zope-4.8.2-py2.7.egg/ZPublisher/httpexceptions.py(30)__call__()
-> return self.application(environ, start_response)
[9]   /home/senaite/dev/buildout-cache/eggs/Zope-4.8.2-py2.7.egg/ZPublisher/WSGIPublisher.py(371)publish_module()
-> response = _publish(request, new_mod_info)
[10]   /home/senaite/dev/buildout-cache/eggs/Zope-4.8.2-py2.7.egg/ZPublisher/WSGIPublisher.py(232)publish()
-> request.processInputs()
[11] > /home/senaite/dev/buildout-cache/eggs/Zope-4.8.2-py2.7.egg/ZPublisher/HTTPRequest.py(738)processInputs()
-> reclist)
[12]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(163)deepcopy()
-> y = copier(x, memo)
[13]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(230)_deepcopy_list()
-> y.append(deepcopy(a, memo))
[14]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(190)deepcopy()
-> y = _reconstruct(x, rv, 1, memo)
[15]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(334)_reconstruct()
-> state = deepcopy(state, memo)
[16]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(163)deepcopy()
-> y = copier(x, memo)
[17]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(257)_deepcopy_dict()
-> y[deepcopy(key, memo)] = deepcopy(value, memo)
[18]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(190)deepcopy()
-> y = _reconstruct(x, rv, 1, memo)
[19]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(334)_reconstruct()
-> state = deepcopy(state, memo)
[20]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(163)deepcopy()
-> y = copier(x, memo)
[21]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(257)_deepcopy_dict()
-> y[deepcopy(key, memo)] = deepcopy(value, memo)
[22]   /home/senaite/dev/Python-2.7/lib/python2.7/copy.py(182)deepcopy()
-> rv = reductor(2)
(Pdb++) key
'conditions'
(Pdb++) item
'<'
(Pdb++) reclist[-1]
{'attachment': '', 'choices': '', 'description': '', 'required': '', 'title': 'Comentarios TGA', 'type': 'text', 'uid': '3939c785fcce47c9a3425d7e1d28aca2'}
(Pdb++) reclist[-2]
{'attachment': '', 'choices': '', 'description': 'Si el an\xc3\xa1lisis no se ajusta al esquema anterior, adjuntar fichero con las condiciones de ensayo', 'required': '', 'title': 'Fichero adjunto', 'type': 'file', 'uid': '3939c785fcce47c9a3425d7e1d28aca2', 'value': <ZPublisher.HTTPRequest.FileUpload object at 0x7fd973388b90>}

What version of Python and Zope/Addons I am using:

Plone 5.2.9 (5217)
CMF 2.6.0
Zope 4.8.2
Python 2.7.15 (default, Jun 2 2020, 22:12:19) [GCC 8.3.0]
PIL 6.2.2 (Pillow)
WSGI: On
Server: waitress 1.4.4

The text was updated successfully, but these errors were encountered:

d-maurer · 2023-01-19T09:20:11Z

Jordi Puiggené wrote at 2023-1-18 02:39 -0800:

A `TypeError: can't pickle cStringIO.StringO objects` is raised when `ZPublisher` processes a request input record with a value that contains a character that needs to be tainted (e.g. "<") and when a record from "file" type has been processed previously.

Tainting is an old security feature for DTML: when form data or cookies are accessed implicitly via the request object, they are (HTML) quoted on DTML rendering if they contain unsafe characters. The tainting logic is incredibly complex and your bug report reveals one of its holes. I have tried to streamline it in "#648", but this is not yet merged (and might have the same problem you have observed). Page templates no longer rely on tainting logic; they quote by default and unquoted rendering must be called for explicitly (and this hopefully is not done for things involving user input). If you do not process untrusted user input with DTML objects, you could disable the tainting logic giving the envvar "ZOPE_DTML_REQUEST_AUTOQUOTE" e.g. the value "0". This should avoid the bug you have reported. Note that Zope's management interface (--> "ZMI") partially still uses DTML. Thus, if you disable tainting, you likely must restrict the use of large parts of the ZMI to trusted users. A good starting point would be to restrict the permission `View management screens` (or similarly spelled) to `Manager`.

d-maurer · 2023-01-20T12:15:10Z

@xispa You could try the version from #1097.

xispa · 2023-01-20T13:05:30Z

Sure, let me try

xispa · 2023-01-20T13:17:31Z

Excellent! 🎊 it works properly now. Thank you very much @d-maurer for your fast response, we really appreciate it!

xispa mentioned this issue Jan 18, 2023

Cannot set conditions with a '<' char when others are from type "file" senaite/senaite.core#2231

Merged

d-maurer mentioned this issue Jan 20, 2023

Sanitize tainting #1097

Merged

d-maurer mentioned this issue Jan 21, 2023

Sanitize tainting (forward port of PR 1097) #1098

Merged

d-maurer closed this as completed in #1098 Jan 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeError: HTTPRequest fails when processing input text records that need tainting and their position is after file type records #1095

TypeError: HTTPRequest fails when processing input text records that need tainting and their position is after file type records #1095

xispa commented Jan 18, 2023

d-maurer commented Jan 19, 2023 via email

d-maurer commented Jan 20, 2023

xispa commented Jan 20, 2023

xispa commented Jan 20, 2023

TypeError: HTTPRequest fails when processing input text records that need tainting and their position is after file type records #1095

TypeError: HTTPRequest fails when processing input text records that need tainting and their position is after file type records #1095

Comments

xispa commented Jan 18, 2023

BUG/PROBLEM REPORT / FEATURE REQUEST

What I did:

What I expect to happen:

What actually happened:

PDB session

What version of Python and Zope/Addons I am using:

d-maurer commented Jan 19, 2023 via email

d-maurer commented Jan 20, 2023

xispa commented Jan 20, 2023

xispa commented Jan 20, 2023