High memory usage when creating subdirectories

[OKT96]

Module

dms

Describe the bug

We are running into a memory spike issue with the dms module and wanted to check if others have experienced the same. When creating a new subdirectory inside a directory that already has many subdirectories and files, memory usage spikes very high during save. In our case the parent directory has ~350 with a total of ~4000 files and ~700mb in size. On small servers with limited RAM this leads to a crash or an MemoryError

To Reproduce

Affected versions: 18.0

Steps to reproduce the behavior:

1. Have a directory containing a large number of subdirectories and files
2. Create a new subdirectory
3. Click save
4. Watch memory spike

Expected behavior
Creating a subdirectory should be lightweight and should not depend on existing content...

[victoralmau]

I've tried to reproduce the error you mentioned, but I haven't been able to.

Can you provide any additional information that might help us identify the problem?

Can you copy the specific log entry related to the MemoryError?

[OKT96]

@victoralmau sure i can share following screenshots:

Traceback (most recent call last):
  File "/opt/odoo/src/odoo/odoo/http.py", line 2167, in _transactioning
    return service_model.retrying(func, env=self.env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/service/model.py", line 157, in retrying
    result = func()
             ^^^^^^
  File "/opt/odoo/src/odoo/odoo/http.py", line 2134, in _serve_ir_http
    response = self.dispatcher.dispatch(rule.endpoint, args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/http.py", line 2382, in dispatch
    result = self.request.registry['ir.http']._dispatch(endpoint)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/addons/base/models/ir_http.py", line 333, in _dispatch
    result = endpoint(**request.params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/http.py", line 754, in route_wrapper
    result = endpoint(self, *args, **params_ok)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/addons/web/controllers/dataset.py", line 36, in call_kw
    return call_kw(request.env[model], method, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/api.py", line 535, in call_kw
    result = getattr(recs, name)(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1002, in onchange
    snapshot0 = RecordSnapshot(record, fields_spec, fetch=(not first_call))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1145, in __init__
    self.fetch(name)
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1158, in fetch
    self[field_name] = {line.id: RecordSnapshot(line, sub_fields_spec) for line in lines}
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1145, in __init__
    self.fetch(name)
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1158, in fetch
    self[field_name] = {line.id: RecordSnapshot(line, sub_fields_spec) for line in lines}
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1145, in __init__
    self.fetch(name)
  File "/opt/odoo/src/odoo/addons/web/models/models.py", line 1160, in fetch
    self[field_name] = self.record[field_name]
                       ~~~~~~~~~~~^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/models.py", line 7083, in __getitem__
    return self._fields[key].__get__(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1309, in __get__
    self.compute_value(recs)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1491, in compute_value
    records._compute_field_value(self)
  File "/opt/odoo/src/odoo/addons/mail/models/mail_thread.py", line 443, in _compute_field_value
    return super()._compute_field_value(field)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/models.py", line 5302, in _compute_field_value
    fields.determine(field.compute, self)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 110, in determine
    return needle(*args)
           ^^^^^^^^^^^^^
  File "/opt/odoo/local/addons-oca-community/addons/dms/models/dms_file.py", line 443, in _compute_mimetype
    binary = base64.b64decode(record.content or "")
                              ^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1309, in __get__
    self.compute_value(recs)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 2616, in compute_value
    super().compute_value(records)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1491, in compute_value
    records._compute_field_value(self)
  File "/opt/odoo/src/odoo/addons/mail/models/mail_thread.py", line 443, in _compute_field_value
    return super()._compute_field_value(field)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/models.py", line 5302, in _compute_field_value
    fields.determine(field.compute, self)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 110, in determine
    return needle(*args)
           ^^^^^^^^^^^^^
  File "/opt/odoo/local/addons-oca-community/addons/dms/models/dms_file.py", line 455, in _compute_content
    if record.content_file:
       ^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1292, in __get__
    value = self.convert_to_cache(rec_origin[self.name], rec, validate=False)
                                  ~~~~~~~~~~^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/models.py", line 7083, in __getitem__
    return self._fields[key].__get__(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1272, in __get__
    recs._fetch_field(self)
  File "/opt/odoo/src/odoo/odoo/models.py", line 4120, in _fetch_field
    self.fetch(fnames)
  File "/opt/odoo/src/odoo/odoo/models.py", line 4158, in fetch
    fetched = self._fetch_query(query, fields_to_fetch)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/models.py", line 4269, in _fetch_query
    field.read(fetched)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 2633, in read
    att.res_id: _encode(human_size(att.file_size)) if bin_size else att.datas
                                                                    ^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1309, in __get__
    self.compute_value(recs)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 2616, in compute_value
    super().compute_value(records)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 1491, in compute_value
    records._compute_field_value(self)
  File "/opt/odoo/src/odoo/odoo/models.py", line 5302, in _compute_field_value
    fields.determine(field.compute, self)
  File "/opt/odoo/src/odoo/odoo/fields.py", line 110, in determine
    return needle(*args)
           ^^^^^^^^^^^^^
  File "/opt/odoo/src/odoo/odoo/addons/base/models/ir_attachment.py", line 238, in _compute_datas
    attach.datas = base64.b64encode(attach.raw or b'')
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MemoryError

Regarding the MemoryError, it likely depends on the number of workers and the limit_memory_hard and limit_memory_soft configurations but either way you should see a memory spike...

I also tested the same process on v16 (we recently upgraded from v16 to v18) and could not reproduce it there as creating and saving a subdirectory was instant

[victoralmau]

Okay, thanks for the screenshots.

In my opinion, this isn't really a DMS error as such (although it could be “fixed” by removing “compute” from the mimetype field https://github.com/OCA/dms/blob/18.0/dms/models/dms_file.py#L101 though it would also be interesting to add “store=true” to other total fields, such as ‘count_directories’ or “count_total_files”—), but rather it has to do with how the process you're trying to perform is carried out:

• You access the form view of a directory that contains 378 subdirectories
• The form view of that directory displays a list view of the 378 subdirectories
• When you try to create a new directory from that list view, it attempts to reload all the data (all the files) from all the subdirectories (378), and that’s why the error occurs.

The way to create a directory or subdirectories—even though you can do it the way you’re trying—is to go to the Directory menu, create a new one (with the appropriate name), and specify the corresponding parent directory—that’s all.

I’ve always thought that the list view showing all the subdirectories of a folder was confusing and problematic, and this proves it.

[OKT96]

@victoralmau thanks a lot for looking into this and for the workaround. I understand the problem now and i guess it is more of a design issue

Maybe one way around this would be to open a new form view page when clicking on 'Add a line' but I am not sure that will work since I assume that this is tied to how the o2m widget works

[pedrobaeza]

o2m fields can open in popup the elements.