...
One of the things we can do against boundless growth of uwsgi is the use of --reload-on-rss <limit>, this kills any worker that exceeds the RSS limit, but results in an empty metadata reply, which is unwanted behaviour. If however, we also supply --lazy, the app is loaded in the worker(s) and the (re)start of each worker then also triggers the reload of metadata. This could be a compromise if the VM is less cpu bound than memory?
Empty Metadata set while refreshing
It turns out pyFF returns an empty metadata set while refreshing, which is unwanted behaviour. The following code, inserted just before the final return in .api#process_handler inspects the validity of the Resource metadata. Having a loadbalancer inspect pyFF and temporarily evicting the server from pool if it receives a 500 could create a stable service.
Code Block |
---|
def process_handler():
...
# Only return request if md is valid?
valid = True
log.debug(f"Resource walk")
for child in request.registry.md.rm.walk():
log.debug(f"Resource {child.url}")
valid = valid and child.is_valid()
if len(request.registry.md.rm) == 0 or not valid:
log.debug(f"Resource not valid")
# 500: The server has either erred or is incapable of performing the requested operation.
raise exc.exception_response(500)
else:
log.debug(f"Resource valid")
return response
|