Formal OHDF Schema Definition
A Look Ahead
In this section, we will cover:
Background
The OHDF schema is a JSON-based schema that uses typed values. This means that any given OHDF file must be formatted in JSON and the typing of each key-value pair in the JSON object must conform to its declared type in the OHDF schema. Nonconformance to these rules will result in an OHDF output that is incompatible with compliant OHDF-based functionality and will produce undefined and erroneous behavior.
Unused Fields
Required Fields
A required field in OHDF is a schema field that must always exist in the file to qualify the file as conforming to the OHDF schema. If a field is required by the OHDF schema, it does not necessarily mean that you have to provide it with meaningful data or correlate it with a data element from another source format. A file can still conform to OHDF by having a required field that has an empty value assigned to it.
Not all fields in the schema have to be filled or even used in an OHDF mapper. In practice, most implementations either provide an empty value (e.g, an empty string, an empty object, etc.) or omit an unused field entirely (i.e., by not defining the key or by passing a value of undefined). Whether or not you should provide an empty value or omit the key for an unused field depends on if the field is declared as required by the OHDF schema. If you have an unused field that is declared as required by the schema, you must define the key and pass it an empty value. If the unused field is not declared as required, you can safely omit the field.
Omitting Schema Fields
Just because you can omit a schema field does not mean that you should!
Always aim to maximize the filled fields in your mapper and only skip a field as a last resort if no applicable equivalent exists in your source data. Consider required fields as priority fields that you should aim to fill out first instead of the bare minimum necessary to create a mapper.
For example, the following required key-value pairs are all considered valid:
{
"platform": {
"name": ""
}
}{
"profiles": [
{
"supports": []
}
]
}Here is a snippet from the previous Twistlock-to-OHDF sample file which shows some empty required fields:
Twistlock-to-OHDF Sample
{
"statistics": {},
"profiles": [
{
"name": "Twistlock Scan",
"title": "Twistlock Project: All / TEST-COLLECTION",
"summary": "Package Vulnerability Summary: 97 Application Compliance Issue Total: 0",
"supports": [],
"attributes": [],
"groups": [],
"status": "loaded",
"sha256": "807c8ef1534bb7f3e428db4a40667a6dd37d89f8a48dc6d1f6bb2426ff53f97f"
}
]
}Breaking Down the Formal OHDF Schema
We can break down the formal OHDF schema in the same manner as before by observing each structure in the hierarchy from a top-down view.
Full OHDF Schema
The following section contains a breakdown of a streamlined version of the OHDF schema.
For the full and up-to-date version, see here.
For the technical implementation of the schema, see here.
Exec JSON
| Field Name | Definition | Type | Required |
|---|---|---|---|
platform | Platform substructure (see Platform). | Platform | Yes |
version | Version of tool generating the file. | string | Yes |
statistics | Statistics substructure (see Statistics). | Statistics | Yes |
profiles | Profiles substructure (see Profiles). | ExecJSONProfile[] | Yes |
passthrough | Passthrough substructure (see Passthrough) | Record<string, unknown> | No |
A basic representation of this structure in JSON is as so:
{
"platform": {},
"version": "",
"statistics": {},
"profiles": [],
"passthrough": {}
}Profiles
| Field Name | Definition | Type | Required |
|---|---|---|---|
name | Name of profile, usually the original security tool. Must be unique. | string | Yes |
version | Version of security tool. | string, null | No |
sha256 | Checksum of the profile (NOTE: AUTOMATICALLY GENERATED BY HDF CONVERTERS, DO NOT POPULATE). | string | Yes |
title | Title of security tool; must be human readable. | string, null | No |
maintainer | Maintainer(s). | string, null | No |
summary | Summary of security data (e.g., the STIG header). | string, null | No |
license | Copyright license. | string, null | No |
copyright | Copyright holder. | string, null | No |
copyright_email | Copyright holder's email. | string, null | No |
supports | Supported platform targets. | SupportedPlatform[] | Yes |
attributes | Inputs/attributes used in scan. | Record<string, unknown>[] | Yes |
groups | Set of descriptions for the control groups (e.g., control IDs). | ControlGroup[] | Yes |
controls | Controls substructure (see Controls). | ExecJSONControl[] | Yes |
status | Status of profile (typically 'loaded'). | string, null | No |
status_message | Reason for the status. | string, null | No |
skip_message | Reason for skipping this profile if skipped. | string, null | No |
depends | Set of dependencies this profile depends on (e.g., an overlay profile is dependent on another profile). | Dependency[], null | No |
parent_profile | Name of the parent profile if this profile is a dependency of another profile. | string, null | No |
description | Description of profile; should be more detailed than summary. | string, null | No |
inspec_version | The version of the Inspec tool used to generate this profile. | string, null | No |
A basic representation of this structure in JSON is as so:
{
"name": "",
"version": "",
"sha256": "",
"title": "",
"maintainer": "",
"summary": "",
"license": "",
"copyright": "",
"copyright_email": "",
"supports": [],
"attributes": [],
"groups": [],
"controls": [],
"status": "",
"status_message": "",
"skip_message": "",
"depends": [],
"parent_profile": "",
"description": "",
"inspec_version": ""
}Knowledge Check
Given a snippet of data, try and identify what information is applicable to the Profiles structure and which fields they would correlate with.
- Data snippet:
GoSec Source Data
{
"Golang errors": {},
"Issues": [
{
"severity": "MEDIUM",
"confidence": "HIGH",
"cwe": {
"id": "22",
"url": "https://cwe.mitre.org/data/definitions/22.html"
},
"rule_id": "G304",
"details": "Potential file inclusion via variable",
"file": "C:\\Users\\AGILLUM\\OneDrive - The MITRE Corporation\\Documents\\Code\\grype-0.34.4\\internal\\file\\tar.go",
"code": "82: \t\tcase tar.TypeReg:\n83: \t\t\tf, err := os.OpenFile(target, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode))\n84: \t\t\tif err != nil {\n",
"line": "83",
"column": "14",
"nosec": false,
"suppressions": null
}
],
"Stats": {
"files": 199,
"lines": 12401,
"nosec": 0,
"found": 7
},
"GosecVersion": "dev"
}- Converted OHDF snippet (may not perfectly align with your answers):
OHDF Converted GoSec Data
"profiles": [
{
"name": "Gosec scanner",
"title": "gosec",
"version": "dev",
"supports": [],
"attributes": [],
"groups": [],
"status": "loaded",
"controls": [
...
],
"sha256": "d0506f4b7715bf8b1cd81a8a87ab8efcce41ebeb2b5ec5fcfb23d3bdd136f48c"
}
]Controls
| Field Name | Definition | Type | Required |
|---|---|---|---|
id | ID of control, used for sorting. Must be unique. | string | Yes |
title | Title of control. | string, null | No |
desc | The overarching description of the control. | string, null | No |
descriptions | Additional descriptions; usually 'check' and 'fix' text for control. | ControlDescription[], null | No |
impact | Security severity of control. On a scale of 0 to 1 and typically increments by 0.1 values. This metric is aligned with the Common Vulnerability Scoring System (CVSS). | number | Yes |
refs | References to external control documentation. | Reference[] | Yes |
tags | Control tags; typically correlate to existing vulnerability/weakness database (e.g., NIST, CVE, CWE). | Record<string, unknown> | Yes |
code | Control source code for code preservation. | string, null | No |
source_location | Location of control within source code. | SourceLocation | Yes |
results | Results substructure (see Results). | ControlResult[] | Yes |
waiver_data | Information on waiver if applicable. | WaiverData, null | No |
attestation_data | Information on attestation if applicable. | AttestationData, null | No |
A basic representation of this structure in JSON is as so:
{
"id": "",
"title": "",
"desc": "",
"descriptions": [],
"impact": 0,
"refs": [],
"tags": {},
"code": "",
"source_location": {},
"results": [],
"waiver_data": {},
"attestation_data": {}
}Knowledge Check
Given a snippet of data, try and identify what information is applicable to the Controls structure and which fields they would correlate with.
- Data snippet:
GoSec Source Data
{
"Golang errors": {},
"Issues": [
{
"severity": "MEDIUM",
"confidence": "HIGH",
"cwe": {
"id": "22",
"url": "https://cwe.mitre.org/data/definitions/22.html"
},
"rule_id": "G304",
"details": "Potential file inclusion via variable",
"file": "C:\\Users\\AGILLUM\\OneDrive - The MITRE Corporation\\Documents\\Code\\grype-0.34.4\\internal\\file\\tar.go",
"code": "82: \t\tcase tar.TypeReg:\n83: \t\t\tf, err := os.OpenFile(target, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode))\n84: \t\t\tif err != nil {\n",
"line": "83",
"column": "14",
"nosec": false,
"suppressions": null
}
],
"Stats": {
"files": 199,
"lines": 12401,
"nosec": 0,
"found": 7
},
"GosecVersion": "dev"
}- Converted OHDF snippet (may not perfectly align with your answers):
OHDF Converted GoSec Data
"controls": [
{
"tags": {
"nist": ["SI-10"],
"cwe": {
"id": "22",
"url": "https://cwe.mitre.org/data/definitions/22.html"
},
"nosec": "",
"suppressions": "",
"severity": "MEDIUM",
"confidence": "HIGH"
},
"refs": [],
"source_location": {},
"title": "Potential file inclusion via variable",
"id": "G304",
"desc": "",
"impact": 0.5,
"results": [
...
]
}
],Results
| Field Name | Definition | Type | Required |
|---|---|---|---|
status | passed/failed status of test (other status include skipped and error). | ControlResultStatus, null | No |
skip_message | Reasoning for if test has skipped status. | string, null | No |
code_desc | Test expectations as defined by control. | string | Yes |
exception | Type of exception if one was thrown. | string, null | No |
backtrace | Backtrace of exception if one was thrown. | string[], null | No |
message | Explanation of test status; typically in the form of expected and actual results. | string, null | No |
resource | Resources used in the test (e.g., Inspec). | string, null | No |
resource_id | The unique ID of the used resources. | string, null | No |
run_time | Overall runtime of test. | number, null | No |
start_time | Starting time of test. | string | Yes |
A basic representation of this structure in JSON is as so:
{
"status": {},
"skip_message": "",
"code_desc": "",
"exception": "",
"backtrace": [],
"message": "",
"resource": "",
"resource_id": "",
"run_time": 0,
"start_time": ""
}Knowledge Review
Given a snippet of data, try and identify what information is applicable to the Results structure and which fields they would correlate with.
- Data snippet:
GoSec Source Data
{
"Golang errors": {},
"Issues": [
{
"severity": "MEDIUM",
"confidence": "HIGH",
"cwe": {
"id": "22",
"url": "https://cwe.mitre.org/data/definitions/22.html"
},
"rule_id": "G304",
"details": "Potential file inclusion via variable",
"file": "C:\\Users\\AGILLUM\\OneDrive - The MITRE Corporation\\Documents\\Code\\grype-0.34.4\\internal\\file\\tar.go",
"code": "82: \t\tcase tar.TypeReg:\n83: \t\t\tf, err := os.OpenFile(target, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode))\n84: \t\t\tif err != nil {\n",
"line": "83",
"column": "14",
"nosec": false,
"suppressions": null
}
],
"Stats": {
"files": 199,
"lines": 12401,
"nosec": 0,
"found": 7
},
"GosecVersion": "dev"
}- Converted OHDF snippet (may not perfectly align with your answers):
OHDF Converted GoSec Data
"results": [
{
"status": "failed",
"code_desc": "82: \t\tcase tar.TypeReg:\n83: \t\t\tf, err := os.OpenFile(target, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode))\n84: \t\t\tif err != nil {\n",
"message": "C:\\Users\\AGILLUM\\OneDrive - The MITRE Corporation\\Documents\\Code\\grype-0.34.4\\internal\\file\\tar.go, line:83, column:14",
"start_time": ""
}
]Platform
| Field Name | Definition | Type | Required |
|---|---|---|---|
name | Name of platform export was run on. | string | Yes |
release | Platform version. | string | Yes |
target_id | Platform target ID (i.e., further identifying information on platform). | string, null | No |
A basic representation of this structure in JSON is as so:
{
"name": "",
"release": "",
"target_id": ""
}Statistics
| Field Name | Definition | Type | Required |
|---|---|---|---|
controls | Break down of control statistics by result status. | StatisticHash, null | No |
duration | Duration of run (in seconds). | number, null | No |
A basic representation of this structure in JSON is as so:
{
"controls": {},
"duration": 0
}Passthrough
There is no formal schema definition for the Passthrough structure. As such, any key-value pair can technically be used in this structure and still be compliant with the OHDF schema. There are, however, some common field naming conventions for certain types of data that are typically placed in Passthrough as defined below:
| Field Name | Definition | Type | Required |
|---|---|---|---|
auxiliary_data | Storage for unused data from the sample file. | Record<string, unknown>[] | Undeclared |
auxiliary_data[].name | Name of auxiliary data source. | string | Undeclared |
auxiliary_data[].data | Auxiliary data itself. | Record<string, unknown> | Undeclared |
raw | Raw data dump of input file. | Record<string, unknown> | Undeclared |
A basic representation of this structure in JSON is as so:
{
"auxiliary_data": [
{
"name": "",
"data": {}
}
],
"raw": {}
}Knowledge Check
Given a snippet of data, try and identify what information is applicable to the Exec JSON, Platform, Statistics, and Passthrough structures and which fields they would correlate with.
- Data snippet:
GoSec Source Data
{
"Golang errors": {},
"Issues": [
{
"severity": "MEDIUM",
"confidence": "HIGH",
"cwe": {
"id": "22",
"url": "https://cwe.mitre.org/data/definitions/22.html"
},
"rule_id": "G304",
"details": "Potential file inclusion via variable",
"file": "C:\\Users\\AGILLUM\\OneDrive - The MITRE Corporation\\Documents\\Code\\grype-0.34.4\\internal\\file\\tar.go",
"code": "82: \t\tcase tar.TypeReg:\n83: \t\t\tf, err := os.OpenFile(target, os.O_CREATE|os.O_RDWR, os.FileMode(header.Mode))\n84: \t\t\tif err != nil {\n",
"line": "83",
"column": "14",
"nosec": false,
"suppressions": null
}
],
"Stats": {
"files": 199,
"lines": 12401,
"nosec": 0,
"found": 7
},
"GosecVersion": "dev"
}- Converted OHDF snippet (may not perfectly align with your answers):
OHDF Converted GoSec Data
{
"platform": { "name": "Heimdall Tools", "release": "2.10.8" },
"version": "2.10.8",
"statistics": {},
"profiles": [],
"passthrough": {
"auxiliary_data": [{ "name": "Gosec", "data": { "Golang errors": {} } }]
}
}A Look Back
In this section, we covered:
Background context for technical aspects of the OHDF schema
- Certain unused/uncorrelated fields in the final OHDF mapping can be omitted/left empty depending on whether the field is qualified as required.
Knowledge Review
Now that we're familiar with the OHDF schema, let's revisit that Twistlock-to-OHDF sample with a specific focus on the fields that the data is assigned to.
Here are some key questions to ask yourself while reading:
Does the information assigned to the schema field make sense?
Does the definition of the schema field match what the assigned data is trying to convey?
Would the data assigned to the schema field be more applicable elsewhere?
Twistlock-to-OHDF Sample
This file has been truncated for pedagogical purposes. For the full file, see here
Twistlock-to-OHDF Sample
{
"platform": {
"name": "Heimdall Tools",
"release": "2.6.29",
"target_id": "registry.io/test:1a7a431a105aa04632f5f5fbe8f753bd245add0a"
},
"version": "2.6.29",
"statistics": {},
"profiles": [
{
"name": "Twistlock Scan",
"title": "Twistlock Project: All / TEST-COLLECTION",
"summary": "Package Vulnerability Summary: 97 Application Compliance Issue Total: 0",
"supports": [],
"attributes": [],
"groups": [],
"status": "loaded",
"controls": [
{
"tags": {
"nist": ["SI-2", "RA-5"],
"cci": ["CCI-002605", "CCI-001643"],
"cveid": "CVE-2021-43529"
},
"refs": [],
"source_location": {},
"title": "CVE-2021-43529",
"id": "CVE-2021-43529",
"desc": "DOCUMENTATION: A remote code execution flaw was found in the way NSS verifies certificates. This flaw allows an attacker posing as an SSL/TLS server to trigger this issue in a client application compiled with NSS when it tries to initiate an SSL/TLS connection. Similarly, a server application compiled with NSS, which processes client certificates, can receive a malicious certificate via a client, triggering the flaw. The highest threat to this vulnerability is confidentiality, integrity, as well as system availability. STATEMENT: The issue is not limited to TLS. Any applications that use NSS certificate verification are vulnerable; S/MIME is impacted as well. Similarly, a server application compiled with NSS, which processes client certificates, can receive a malicious certificate via a client. Firefox is not vulnerable to this flaw as it uses the mozilla::pkix for certificate verification. Thunderbird is affected when parsing email with the S/MIME signature. Thunderbird on Red Hat Enterprise Linux 8.4 and later does not need to be updated since it uses the system NSS library, but earlier Red Hat Enterprise Linux 8 extended life streams will need to update Thunderbird as well as NSS. MITIGATION: Red Hat has investigated whether a possible mitigation exists for this issue, and has not been able to identify a practical example. Please update the affec",
"impact": 0.9,
"code": "{\n \"id\": \"CVE-2021-43529\",\n \"status\": \"affected\",\n \"cvss\": 9.8,\n \"description\": \"DOCUMENTATION: A remote code execution flaw was found in the way NSS verifies certificates. This flaw allows an attacker posing as an SSL/TLS server to trigger this issue in a client application compiled with NSS when it tries to initiate an SSL/TLS connection. Similarly, a server application compiled with NSS, which processes client certificates, can receive a malicious certificate via a client, triggering the flaw. The highest threat to this vulnerability is confidentiality, integrity, as well as system availability. STATEMENT: The issue is not limited to TLS. Any applications that use NSS certificate verification are vulnerable; S/MIME is impacted as well. Similarly, a server application compiled with NSS, which processes client certificates, can receive a malicious certificate via a client. Firefox is not vulnerable to this flaw as it uses the mozilla::pkix for certificate verification. Thunderbird is affected when parsing email with the S/MIME signature. Thunderbird on Red Hat Enterprise Linux 8.4 and later does not need to be updated since it uses the system NSS library, but earlier Red Hat Enterprise Linux 8 extended life streams will need to update Thunderbird as well as NSS. MITIGATION: Red Hat has investigated whether a possible mitigation exists for this issue, and has not been able to identify a practical example. Please update the affec\",\n \"severity\": \"critical\",\n \"packageName\": \"nss-util\",\n \"packageVersion\": \"3.67.0-7.el8_5\",\n \"link\": \"https://access.redhat.com/security/cve/CVE-2021-43529\",\n \"riskFactors\": [\n \"Remote execution\",\n \"Attack complexity: low\",\n \"Attack vector: network\",\n \"Critical severity\",\n \"Recent vulnerability\"\n ],\n \"impactedVersions\": [\n \"*\"\n ],\n \"publishedDate\": \"2021-12-01T00:00:00Z\",\n \"discoveredDate\": \"2022-05-18T12:24:22Z\",\n \"layerTime\": \"2022-05-16T23:12:25Z\"\n}",
"results": [
{
"status": "failed",
"code_desc": "Package \"nss-util\" should be updated to latest version above impacted versions [\"*\"]",
"message": "Expected latest version of \"nss-util\"\nDetected vulnerable version \"3.67.0-7.el8_5\" of \"nss-util\"",
"start_time": "2022-05-18T12:24:22Z"
},
{
"status": "failed",
"code_desc": "Package \"nss-sysinit\" should be updated to latest version above impacted versions [\"*\"]",
"message": "Expected latest version of \"nss-sysinit\"\nDetected vulnerable version \"3.67.0-7.el8_5\" of \"nss-sysinit\"",
"start_time": "2022-05-18T12:24:22Z"
},
{
"status": "failed",
"code_desc": "Package \"nss\" should be updated to latest version above impacted versions [\"*\"]",
"message": "Expected latest version of \"nss\"\nDetected vulnerable version \"3.67.0-7.el8_5\" of \"nss\"",
"start_time": "2022-05-18T12:24:22Z"
},
{
"status": "failed",
"code_desc": "Package \"nss-softokn\" should be updated to latest version above impacted versions [\"*\"]",
"message": "Expected latest version of \"nss-softokn\"\nDetected vulnerable version \"3.67.0-7.el8_5\" of \"nss-softokn\"",
"start_time": "2022-05-18T12:24:22Z"
},
{
"status": "failed",
"code_desc": "Package \"nss-softokn-freebl\" should be updated to latest version above impacted versions [\"*\"]",
"message": "Expected latest version of \"nss-softokn-freebl\"\nDetected vulnerable version \"3.67.0-7.el8_5\" of \"nss-softokn-freebl\"",
"start_time": "2022-05-18T12:24:22Z"
}
]
}
],
"sha256": "807c8ef1534bb7f3e428db4a40667a6dd37d89f8a48dc6d1f6bb2426ff53f97f"
}
],
"passthrough": {
"auxiliary_data": [
{
"name": "Twistlock",
"data": {
"results": [
{
"id": "sha256:b1c0237ebd29860066656372da10d8d7da33b34715986f74b3d5a7e4ba060d1b",
"distro": "Red Hat Enterprise Linux release 8.6 (Ootpa)",
"distroRelease": "RHEL8",
"digest": "sha256:c0274cb1e0c6e92d2ccc1e23bd19b7dddedbaa2da26861c225d4ad6cec5047f4",
"packages": [
{
"type": "os",
"name": "nss-util",
"version": "3.67.0-7.el8_5",
"licenses": ["MPLv2.0"]
},
{
"type": "os",
"name": "nss-softokn",
"version": "3.67.0-7.el8_5",
"licenses": ["MPLv2.0"]
},
{
"type": "os",
"name": "nss",
"version": "3.67.0-7.el8_5",
"licenses": ["MPLv2.0"]
},
{
"type": "os",
"name": "nss-softokn-freebl",
"version": "3.67.0-7.el8_5",
"licenses": ["MPLv2.0"]
},
{
"type": "os",
"name": "nss-sysinit",
"version": "3.67.0-7.el8_5",
"licenses": ["MPLv2.0"]
}
],
"applications": [
{
"name": "asp.net-core",
"version": "5.0.17",
"path": "/usr/lib64/dotnet/dotnet"
},
{
"name": ".net-core",
"version": "5.0.17",
"path": "/usr/lib64/dotnet/dotnet"
}
],
"complianceScanPassed": true,
"vulnerabilityScanPassed": true,
"history": [
{
"created": "2022-05-03T08:38:31Z"
},
{
"created": "2022-05-03T08:39:27Z"
},
{
"created": "2022-05-16T23:12:02Z",
"instruction": "ARG GOPROXY=http://nexus-repository-manager.nexus-repository-manager.svc.cluster.local:8081/repository/goproxy/ HTTP_PROXY=http://localhost:3128 http_proxy=http://localhost:3128"
},
{
"created": "2022-05-16T23:12:02Z",
"instruction": "ARG GOPROXY=http://nexus-repository-manager.nexus-repository-manager.svc.cluster.local:8081/repository/goproxy/ GOSUMDB=sum.golang.org http://nexus-repository-manager.nexus-repository-manager.svc.cluster.local:8081/repository/gosum HTTP_PROXY=http://localhost:3128 http_proxy=http://localhost:3128"
}
],
"scanTime": "2022-05-18T12:24:32.855444532Z",
"scanID": "6284e580d9600f8d0db159e2"
}
],
"consoleURL": "https://twistlock.test.net/#!/monitor/vulnerabilities/images/ci?search=sha256%3Ab1c0237ebd29860066656372da10d8d7da33b34715986f74b3d5a7e4ba060d1b"
}
}
]
}
}