Changeset View
Changeset View
Standalone View
Standalone View
swh/loader/mercurial/objects.py
| Show First 20 Lines • Show All 249 Lines • ▼ Show 20 Lines | class SelectiveCache(OrderedDict): | ||||
| """Special cache for storing past data upon which new data is known to be | """Special cache for storing past data upon which new data is known to be | ||||
| dependent. Optional hinting of how many instances of which keys will be | dependent. Optional hinting of how many instances of which keys will be | ||||
| needed down the line makes utilization more efficient. And, because the | needed down the line makes utilization more efficient. And, because the | ||||
| distance between related data can be arbitrarily long and the data | distance between related data can be arbitrarily long and the data | ||||
| fragments can be arbitrarily large, a disk-based secondary storage is used | fragments can be arbitrarily large, a disk-based secondary storage is used | ||||
| if the primary RAM-based storage area is filled to the designated capacity. | if the primary RAM-based storage area is filled to the designated capacity. | ||||
| Storage is occupied in three phases: | Storage is occupied in three phases: | ||||
| 1) The most recent key/value pair is always held, regardless of other | 1) The most recent key/value pair is always held, regardless of other | ||||
| factors, until the next entry replaces it. | factors, until the next entry replaces it. | ||||
| 2) Stored key/value pairs are pushed into a randomly accessible | 2) Stored key/value pairs are pushed into a randomly accessible | ||||
| expanding buffer in memory with a stored size function, maximum size | expanding buffer in memory with a stored size function, maximum size | ||||
| value, and special hinting about which keys to store for how long | value, and special hinting about which keys to store for how long | ||||
| optionally declared at instantiation. | optionally declared at instantiation. | ||||
| 3) The in-memory buffer pickles into a randomly accessible disk-backed | 3) The in-memory buffer pickles into a randomly accessible disk-backed | ||||
| secondary buffer when it becomes full. | secondary buffer when it becomes full. | ||||
| Occupied space is calculated by default as whatever the len() function | Occupied space is calculated by default as whatever the len() function | ||||
| returns on the values being stored. This can be changed by passing in a new | returns on the values being stored. This can be changed by passing in a new | ||||
| size_function at instantiation. | size_function at instantiation. | ||||
| The cache_hints parameter is a dict of key/int pairs recording how many | The cache_hints parameter is a dict of key/int pairs recording how many | ||||
| subsequent fetches that particular key's value should stay in storage for | subsequent fetches that particular key's value should stay in storage for | ||||
| before being erased. If you provide a set of hints and then try to store a | before being erased. If you provide a set of hints and then try to store a | ||||
| key that is not in that set of hints, the cache will store it only while it | key that is not in that set of hints, the cache will store it only while it | ||||
| is the most recent entry, and will bypass storage phases 2 and 3. | is the most recent entry, and will bypass storage phases 2 and 3. | ||||
| """ | """ | ||||
| DEFAULT_SIZE = 800*1024*1024 # bytes or whatever | DEFAULT_SIZE = 800*1024*1024 # bytes or whatever | ||||
| def __init__(self, max_size=None, cache_hints=None, | def __init__(self, max_size=None, cache_hints=None, | ||||
| size_function=None, filename=None): | size_function=None, filename=None): | ||||
| """args: | """ | ||||
| args: | |||||
| max_size: integer value indicating the maximum size of the part | max_size: integer value indicating the maximum size of the part | ||||
| of storage held in memory | of storage held in memory | ||||
| cache_hints: dict of key/int pairs as described in the class | cache_hints: dict of key/int pairs as described in the class | ||||
| description | description | ||||
| size_function: callback function that accepts one parameter and | size_function: callback function that accepts one parameter and | ||||
| returns one int, which should probably be the | returns one int, which should probably be the calculated | ||||
| calculated size of the parameter | size of the parameter | ||||
| """ | """ | ||||
| self._max_size = max_size or SelectiveCache.DEFAULT_SIZE | self._max_size = max_size or SelectiveCache.DEFAULT_SIZE | ||||
| self._disk = None | self._disk = None | ||||
| if size_function is None: | if size_function is None: | ||||
| self._size_function = sys.getsizeof | self._size_function = sys.getsizeof | ||||
| else: | else: | ||||
| self._size_function = size_function | self._size_function = size_function | ||||
| self._latest = None | self._latest = None | ||||
| ▲ Show 20 Lines • Show All 120 Lines • Show Last 20 Lines | |||||