Cohasset Associates Report

2.1 Non-Rewriteable, Non-Erasable Record Format

2.1.1 Compliance Requirement [SEC 17a-4(f)(2)(ii)(A)]

As set forth in Section III(B) of the 2001 Interpretive Release, this requirement “is designed to ensure that electronic records are capable of being accurately reproduced for later reference by maintaining the records in an unalterable form [for the required retention period].”

The following statement in the 2003 Interpretive Release further clarifies that certain implementations of rewriteable and erasable media, such as magnetic disk or magnetic tape, meet the requirements of a non-rewriteable and non-erasable recording environment provided: (a) the storage solution delivers the prescribed functionality, and (b) the functionality is delivered via appropriate integrated control codes for the SEC designated retention period associated with the stored records.

A broker-dealer would not violate the requirement in paragraph (f)(2)(ii)(A) of the rule if it used an electronic storage system that prevents the overwriting, erasing or otherwise altering of a record during its required retention period through the use of integrated hardware and software control codes. [emphasis added]

Further, Section IV of the 2003 Interpretive Release places requirements on the storage system for retaining records beyond the SEC-established retention period when certain circumstances occur, such as a subpoena or legal hold:

Moreover, there may be circumstances (such as receipt of a subpoena) where a broker-dealer is required to maintain records beyond the retention periods specified in Rule 17a-4 or other applicable Commission rules. Accordingly, a broker-dealer must take appropriate steps to ensure that records are not deleted during periods when the regulatory retention period has lapsed but other legal requirements mandate that the records continue to be maintained, and the broker-dealer’s storage system must allow records to be retained beyond the retentions periods specified in Commission rules. [emphasis added]

This statement by the SEC clarifies that the storage system must have the capability to retain records beyond the retention period established at the time of initial recording when required for legal matters, external investigations or audits, or other similar circumstances.

2.1.2 Compliance Assessment

It is Cohasset’s opinion that the current features of MinIO Object Storage, with Object Lock enabled on the bucket and record objects stored in Compliance Mode, meet this SEC requirement to retain records in non-rewriteable, non-erasable format for time-based³ retention periods and any applied legal holds, when (a) properly configured, as described in Section 2.1.3, and (b) the considerations described in Section 2.1.4 are satisfied.

2.1.3 MinIO Object Storage Capabilities

Overview

• To meet the non-rewriteable, non-erasable requirements of SEC Rule 17a-4(f), a record requiring time-based retention, must (a) be stored in a Bucket with the Object Lock feature enabled, (b) have the Object Lock mode set to Compliance (hereinafter Compliance Mode), and (c) have a Retain Until Date applied to each record object (version). With this configuration, MinIO disables all application programming interfaces (APIs) that could potentially alter or prematurely delete the record object and immutable metadata.
• Object Lock mode may be set, either (a) explicitly by the user/API, or (b) implicitly by inheriting the default values configured for the bucket.
• In addition to Compliance Mode, MinIO Object Lock offers Governance Mode, which allows authorized users to remove Object Lock from an object. Therefore, only Compliance Mode meets the requirements of the Rule.
• When litigation or a subpoena requires record objects to be placed on hold, which could entail retention beyond the assigned retention period, a Legal Hold status may be applied to the record objects. This prohibits deletion of the record objects until the Legal Hold status is removed.

The fundamental features of MinIO Object Storage prevent changes or modifications to record objects and associated immutable metadata, once stored. Further, when the above configurations and settings are applied:

• The Object Lock feature cannot be removed from a Bucket that contains record objects.
• When Compliance Mode is set, on the record object, it cannot be removed, and prevents the specified Retain Until Date from being shortened or removed. It can only be extended, if necessary.
• The record object and its immutable metadata cannot be modified, overwritten or deleted until both (a) the Retain Until Date has expired and (b) the Legal Hold status is removed. MinIO Bucket and User Policy Configurations

MinIO Bucket and User Policy Configurations

• For each Bucket retaining record objects requiring compliance with SEC Rule 17a-4(f), the Object Lock feature must be enabled when the bucket is created. Enabling Object Lock automatically enables the Versioning feature. Once the Object Lock feature is enabled, it cannot be suspended or disabled.
• Optionally, Bucket default values may be set for (a) Object Lock mode, (b) Default retention duration (e.g., 6 Years), and (c) Minimum and Maximum retention durations. Once configured, these defaults automatically apply to each stored record object and metadata, unless retention controls are explicitly transmitted with the record object.
  • Object Lock mode and the Default retention period must be configured together or neither default can be configured.
    • The default Object Lock mode may be set to must be set to Compliance, for record objects requiring compliance with SEC Rule 17a-4(f)
    • The Default retention duration is added to the storage date to calculate the record object’s Retain Until Date. (See section Record Object Definition and Retention Controls, for more information).
  • The Bucket Minimum and Maximum retention durations only apply to anonymous users. Known users are governed by the User Policy Minimum and Maximum retention durations, described in the following paragraph.
• User Policies, configured via the S3 API, define a set of permissions that grant access to actions and resources in MinIO Object Storage. Optionally, a User Policy may constrain the user (e.g., source application) to an explicit minimum and maximum range for the applied Retain Until Dates.
  • Since the Minimum and Maximum retention range is set through the User Policy, each permissioned user of a Bucket may be bound by a different Minimum and Maximum range. If a user attempts to set retention outside of this range, the request is denied.
    • Authorized users may change the Minimum and Maximum range at any time. The updated Minimum and Maximum applies to new record objects and does not apply to previously stored record objects.
• IMPORTANT NOTE: When a record object is not transmitted with an explicit Retain Until Date, either (a) the object is stored without any retention controls (if no Bucket defaults were configured) or (b) the Bucket Default retention period is applied to the record object even if the default is outside the Minimum and Maximum range for the user, thus overriding the policy. Therefore, setting the Default retention period requires careful planning to assure an appropriate Retain Until Date is set, when an explicit Retain Until Date is not transmitted with the record object.

Record Object Definition and Retention Controls

• Each record object is comprised of:
  • Complete content of the record object,
  • Immutable Metadata, which includes, but is not limited to, unique object Key name, version identifier (VersionID), creation/storage (last modified) date and time, object size, and user-defined custom metadata (key-value pairs), and
  • Mutable Metadata, which includes Retain Until Date, Object Lock mode and Legal Hold status
• Each record object has a separate Retain Until Date and Object Lock mode either transmitted with it or inherited from Bucket default values. (REMINDER: The term record object is defined as a version of a record object.)
• The Object Lock mode can be set to one of three options (null, Governance or Compliance) for a given record object and its metadata; only Compliance Mode meets the requirements of SEC Rule 17a-4(f).
• 1.Object Lock mode set to Compliance, assures the following retention controls:
  • The Retain Until Date may be extended to a future date but cannot be shortened or cleared by any user, including the account root user.
  • The Object Lock mode cannot be changed to Governance or cleared (null) by any user, including the account root user.
• 2.Object Lock mode set to Governance, permits clearing the Object Lock mode and the Retain Until Date. As a result, Governance is disallowed for record objects required to comply with the Rule.
• 3.Object Lock mode may be null (blank), which does not apply any retention controls and, therefore, is disallowed for records required to comply with the Rule.
• The following MinIO Object Store features prevent modification, overwrite and deletion, until eligible:
  • The fundamental capabilities of Compliance Mode, when enabled, immutably stores record objects and immutable metadata.
  • The Versioning feature ensures record objects are not overwritten; instead, a new version is created.
  • Each record object is protected from deletion when either:
    • The Retain Until Date of the record object has a future date, or
    • The Legal Hold status of the record object is enabled (On).
  • For record objects stored in Compliance Mode, the Retain Until Date may be extended to a future date but cannot be shortened or cleared, by any user, including the account root user.
• To apply Compliance Mode and a Retain Until Date to the record object, as required to comply with the Rule, either: (a) the source application transmits Compliance Mode and an explicit Retain Until Date with a record object, or (b) Bucket defaults apply Compliance Mode and a Default retention duration for record objects that are transmitted without retention values,.
• A record object may be copied between Buckets, resulting in the creation of a new copy with its own unique metadata. The copy does not retain the original record object’s Retain Until Date, Object Lock mode and Legal Hold status; therefore, the attributes need to be set via Bucket defaults or explicitly. The original record object and metadata will remain, unaltered, in the original Bucket.
• The following user actions are rejected, when Object Lock mode is set to Compliance:
  • Shorten or remove a record object’s Retain Until Date in Compliance Mode.
  • Change the Object Lock mode from Compliance to Governance or from Compliance to null (blank).
  • Delete a record object, by VersionID, before the Retain Until Date has passed (expired).

Legal Hold

When litigation, regulatory investigation, or a subpoena requires record objects to be placed on hold, which could entail retention beyond the assigned retention period, the regulated entity must ensure the subject record objects are protected for the duration of the legal hold.

• The Legal Hold status (On/Off) may be applied to any record object stored in a Bucket with the Object Lock feature enabled.
  • Each record object version includes a separate Legal Hold status attribute.
  • The Legal Hold status is independent of the record object’s Retain Until Date and Object Lock mode; therefore, a Legal Hold status may be applied to any record object in a Bucket with the Object Lock feature enabled, including record objects without a Retain Until Date and Object Lock mode.
  • When the Legal Hold status is set (On), it prohibits deleting the record object until the Legal Hold status is removed (Off).
  • When the Legal Hold status is cleared (Off), this attribute no longer mandates preservation of the record object; however, the retention controls continue to apply to the record object.
• The Legal Hold status for a record object can be verified by either:
  (a) using Stat command to view the metadata for the object or
  (b) issuing ‘get-object-legal hold’ through the S3 API.

Managing Versions

• Enabling the Object Lock feature for a Bucket automatically enables the Versioning feature.
• When the versioning feature is enabled, each version of the record object is separately managed, in accordance with the following controls:
  1. 1.A new version is created when the file contents or metadata are changed or when a new file (with the same Key name) is uploaded.
  2. 2.A new version is not created when retention controls (Object Lock mode and Retain Until date) are applied or when the Legal Hold status is applied or removed for a stored version of the record object.
  3. 3.The retention controls use the version creation/storage date and time:
     • When the Bucket Default retention duration is applied to the version, it is added to the creation/storage date and time to calculate the Retain Until date for the version.
     • When a Minimum and Maximum range applies, the version creation/storage date and time is used for the validation.
  4. 4.Deleting a record object version by Key name without specifying a VersionID creates a ‘delete marker’, which is then considered the most recent version. The ‘delete marker’ does not affect the stored versions of the record. The ‘delete marker’ may be deleted in the future.
  5. 5.When attempting to delete a record object by VersionID, Compliance Lock protections apply, and only eligible versions are deleted. An error message is communicated, and the deletion operation fails, if the version is ineligible for deletion.

Deletion Controls

• The Retain Until Date and Legal Hold status determine if the record object is eligible for deletion (eligibility for deletion does not cause automatic deletion). The following criteria must be met for a record object to be eligible for deletion:
  • The Retain Until Date must have expired (date prior to current date).
  • The Legal Hold status must be clear (Off).
• The Bucket cannot be deleted, until it is empty.

Clock Management

• To meet the requirements of the Rule, Cohasset asserts that every system clock must synchronize to an external time server, e.g., a network time protocol (NTP) clock. The MinIO Object Storage system must be configured to enable NTP and regularly check the time of the external source (NTP) and resynchronize time.
  • When Object Lock is enabled, MinIO Object Storage prohibits updating the system clock locally. These controls prevent or correct any inadvertent or intentional administrative modifications of the time clock, which could allow for premature deletion of record objects.

Security

• MinIO Object Storage is designed to meet Enterprise security and compliance requirements. MinIO Object Storage supports the following server-side encryption schemes to protect data at rest and in motion:
  • Encryption is supported using AES-256-GCM and ChaCha20-Poly1305.
  • Encrypted objects are tamper-proofed with AEAD server-side encryption.
  • MinIO Object Storage is compatible with commonly used Key Management solutions (e.g., HashiCorp Vault).
  • MinIO Object Storage uses a key management-system (KMS) to support SSE-S3. If a client requests SSE-S3, or auto-encryption is enabled, the MinIO Object Storage server encrypts each object with a unique object key which is protected by a master key managed by the KMS.
  • MinIO Object Storage may be configured to protect data in-transit (data traveling to and from MinIO Object Storage) may be protected using Secure Sockets Layer (SSL).
  • Roles-based Security (RBAC) is employed by MinIO Object Storage. The user is identified by access key and policy to allow S3 API (Application Programming Interface) calls. The permissions for each user are controlled through User Policies.

2.1.4 Additional Considerations

To assure compliance with the non-erasable and non-rewritable requirements of the SEC Rule, the regulated entity is responsible for:

• Assigning permissions required to manage the retention controls and property configuring the User Policies and MinIO Object Store Buckets that will retain regulated records. NOTE: Cohasset recommends setting Bucket defaults: (a) Object Lock in Compliance Mode and (b) an appropriate retention period that complies with the regulatory retention requirements.
• Optionally setting Minimum and Maximum retention durations for the Bucket (which apply to anonymous users) or for User Policies, to validate the Retain Until Date applied to each record object.
• Applying the retention controls to each record object that is required for regulatory compliance.
  • Setting the Object Lock mode to Compliance
  • Applying a Retain Until Date that meets regulatory retention requirements
  Cohasset recommends that retention controls be applied within 24 hours of storing a record object required for compliance with the Rule:
• Setting a Legal Hold status to On, when required, to preserve record objects for legal matters, government investigations, external audits and other similar circumstances. NOTE: The Legal Hold status should be set to Off, when preservation is no longer required.
• Limiting the creation and management of ‘delete markers.’ NOTE: Cohasset recommends always specifying the VersionID in delete requests.
• Storing record objects requiring event-based⁴ retention periods in a separate compliance system, since MinIO Object Storage does not currently support event-based retention periods.
• Setting appropriate security controls to (1) restrict network ports and protocol access, (2) establish roles-based access, and (3) encrypt data in transit and while at rest.

2.2Accurate Recording Process

2.2.1 Compliance Requirement [SEC 17a-4(f)(2)(ii)(B)]

The intent of this requirement is to ensure both the accuracy and quality of the recording process such that the records read from the storage media are precisely the same as those that were recorded. This requirement includes both a quality verification of the recording process and post-recording verification processes.

2.2.2 Compliance Assessment

Cohasset affirms that the capabilities of MinIO Object Storage, in conjunction with the inherent capabilities of advanced magnetic storage technology, meet this SEC requirement for accurate recording and post-recording verification.

2.2.3 MinIO Object Storage Capabilities

MinIO Object Storage has a combination of recording and post-recording verification processes, which are described in the following subsections.

Recording Process:

• An MD5 checksum must be transmitted with the record object. The record object will be stored only if the MD5 checksum value calculated by MinIO Object Storage matches the uploaded checksum. If it does not match, an error is reported, and the record object must be re-uploaded.
• MinIO Object Storage utilizes advanced electronic recording technology which applies a combination of checks and balances to assure that record objects are written in a high quality and accurate manner.
• Each record object and associated metadata is protected with erasure code and a bit rot hash. Post-Recording Verification Process:

Post-Recording Verification Process:

• MinIO Object Storage uses erasure coding to split the record object into data segments (chunks) and assigns a checksum to each segment. Integrity of the record object is validated by comparing the checksum of each segment during read, to ensure that an accurate record object is delivered.
• MinIO Object Storage employs a background healing process that scans the data segments, checking and correcting errors. If a segment is corrupt, meaning the checksum value is invalid, an automatic recovery process is initiated to rebuild the segment from the other valid segments.
  • MinIO Object Storage allows the administrator to manually heal (correct) errors, although auto-healing is utilized when possible.

2.2.4 Additional Considerations

There are no additional considerations related to this requirement.

2.3Serialize the Original and Duplicate Units of Storage Media

2.3.1 Compliance Requirement [SEC 17a-4(f)(2)(ii)(C)]

This requirement, according to Section III(B) of the SEC’s 2001 Interpretive Release, “is intended to ensure both the accuracy and accessibility of the records by indicating the order in which records are stored, thereby making specific records easier to locate and authenticating the storage process.”

When the SEC Rule was issued in 1997, this requirement was thought to be more pertinent to tracking the individual units of removable media related to micrographic or optical storage. This requirement for non-unitized electronic storage may be satisfied by capturing and storing immutable metadata, associated with each electronic record, to uniquely identify the record and the date and time of recording.

2.3.2 Compliance Assessment

It is Cohasset’s opinion that the capabilities of MinIO Object Storage meet this SEC requirement to serialize the original and duplicate records.

2.3.3MinIO Object Storage Capabilities

• Each record object is serialized in MinIO Object Storage Buckets using a combination of: (a) unique object Key name, (b) VersionID, and (c) creation/storage date and time stamp. These attributes are immutable.
  • The object name must be unique within the Bucket.
  • The VersionID is automatically assigned.
  • The creation/storage date (last modified) date and time is system-defined, immutable, and stored with each record object.
• This combination of unique object Key name, VersionID, and creation/storage date and time serializes each record object in both space and time.

2.3.4Additional Considerations

There are no additional considerations related to this requirement.

2.4Capacity to Download Indexes and Records

This requirement necessitates an adequate capacity to readily download records and associated indexes, in a format and on a medium acceptable under the Rule and as specified by the SEC or self-regulatory organization. This allows the SEC or self-regulatory organizations to take possession of the downloaded records and indexes.

2.4.2 Compliance Assessment

It is Cohasset’s opinion that the capabilities of MinIO Object Storage meet this SEC requirement to readily download records and indexes (metadata attributes), when the considerations described in Section 2.4.4 are addressed.

2.4.3 MinIO Object Storage Capabilities

• MinIO Object Storage users with administrative permissions can find objects and associated metadata using Linux commands which have been expanded to cover governance features.
  • Search functionality is limited to the name of the record object, VersionID or list all objects.
  • Metadata is not searchable; however, the stat command can be utilized to list metadata.
  • For advanced search capabilities, metadata can be inserted into a custom database and then used to filter basic metadata and head attributes
• Record objects and indexes (metadata attributes) may be downloaded using the S3 API. The following capabilities support the capacity to download record objects and index data (metadata attributes) using the S3 API:
  • List record objects including all versions, including delete markers, in a Bucket (selection criteria based on name may be refined to find and return a subset of the objects in a Bucket).
  • Download selected record objects and the associated indexes (metadata attributes) to a designated storage location.

2.4.4 Additional Considerations

The regulated entity is responsible for (a) ensuring record objects with delete markers are included in the search results, (b) authorizing user permissions, (c) maintaining hardware and software to access MinIO Object Storage, (d) assuring that the regulator, self-regulatory organization or designated examining authority receive downloads of the requested record objects and associated indexes (metadata attributes), in the requested format and medium.

2.5 Duplicate Copy of the Records Stored Separately

2.5.1 Compliance Requirement [SEC 17a-4(f)(3)(iii)]

The intent of this requirement is to provide an alternate source for accessing the records, should the primary source be compromised, i.e., lost or damaged.

Note: A duplicate copy is defined as a persistent copy that allows the complete and accurate record to be reestablished from data stored on a compliant storage system or medium. Whereas, a backup copy is defined as a non-persistent copy that is overwritten as it is rotated on a periodic basis, resulting in a much shorter retention period than the original.

2.5.2 Compliance Assessment

Cohasset asserts that MinIO Object Storage meets this SEC requirement for a persistent duplicate copy of the record objects, when (a) properly configured, as described in Section 2.5.3, and (b) the considerations described in Section 2.5.4 are satisfied.

2.5.3 MinIO Object Storage Capabilities

There are two options for meeting the conditions of this requirement to separately store a duplicate copy.

Duplicate Using Erasure Coding

• MinIO Object Storage uses erasure coding (EC) to store data segments (chunks) of record objects redundantly across multiple nodes. In the event of a disk or node failure, the original record object can be regenerated from redundant data segments.
  • MinIO creates erasure-coding sets of 4 to 16 drives per set.
  • Each object is written to a single erasure coding set, and therefore, is spread over no more than 16 drives.
  • A record object is regenerated from the erasure encoded data.
  • The erasure coded data segments are retained for the full retention period and any applied Legal Holds.

Duplicate Using Mirror

• Duplicate copies may be stored using the Mirror functionality, which provides continuous synchronization at the Bucket level.
  • When mirroring is configured it copies the default lock configuration from the source bucket.
  • Watch flag identifies changes in the source bucket and copies the changes to the destination bucket. If the destination bucket is unavailable, it will continue to retry until the copy is completed.
  • Scheduled jobs collect source bucket changes not processed and copies the changes to the destination bucket to ensure synchronization.
  • When Object Lock or Legal Hold is added to record objects in the source bucket, an event is triggered to apply these controls to the object on the destination server.

2.5.4 Additional Considerations

The regulated entity is responsible for the following.

• When using Mirror functionality, ensure that a scheduled job is configured to run, at a minimum daily, to collect any non-replicated changes on the source bucket and copy the changes to the destination bucket.
• When relying exclusively on erasure coding for the duplicate copy, Cohasset recommends the regulated entity configure the system such that the storage pools are equally distributed across three or more geographically dispersed data centers.

---------

3 Time-based retention periods require records to be retained for a specified contiguous period of time from the date and time created and stored.

4 Event-based or event-time-based retention periods require records to retained indefinitely until a specified event occurs (e.g., a contract expires, or an employee terminates), after which the record is retained for a fixed final retention period.