Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 26 additions & 36 deletions scripts/snapshots/report_cel_detailed.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# cloudformation-validate vs cfn-lint - Parity Report

> Generated: 2026-09-02 18:43:04
> Generated: 2026-09-03 11:08:29
> Engine: **cel**
> Detail level: **detailed**
> Matching: `(rule_id, resource_id, path)` two-pass with `(rule_id, resource_id)` fallback + aliases
Expand All @@ -23,22 +23,22 @@
| Metric | Value |
|--------|------:|
| True Positives | 4053 |
| False Positives (engine bugs) | 89 |
| False Positives (engine bugs) | 87 |
| Engine Extra (correct, cfn-lint gap) | 8257 |
| False Negatives (engine misses) | 356 |
| Precision | 97.85% |
| Precision | 97.90% |
| Recall | 91.93% |
| F1 | 94.80% |
| F1 | 94.82% |
| Unique rules detected | 238 |
| Perfect templates | 494/665 |
| Perfect templates | 495/665 |
| Location mismatches (matched pairs) | 9 |

### By Severity

| Severity | TP | FP | EE | FN | Precision | Recall |
|----------|---:|---:|---:|---:|----------:|-------:|
| Fatal | 447 | 14 | 81 | 145 | 96.96% | 75.51% |
| Error | 856 | 63 | 12 | 140 | 93.14% | 85.94% |
| Fatal | 447 | 13 | 81 | 145 | 97.17% | 75.51% |
| Error | 856 | 62 | 12 | 140 | 93.25% | 85.94% |
| Warning | 2080 | 0 | 371 | 61 | 100.00% | 97.15% |
| Info | 670 | 12 | 7793 | 10 | 98.24% | 98.53% |

Expand Down Expand Up @@ -1040,7 +1040,7 @@ but found another document
- **W6001** → `Outputs.ImportedValue.Value.Fn::ImportValue` L39 in `good_output_value_string_yaml`
> The output value {'Fn::ImportValue': 'SomeExportedName'} is an import from another output

## False Positives - 89 extra findings across 15 rules
## False Positives - 87 extra findings across 15 rules

These are diagnostics the engine reports but cfn-lint does not expect (potential bugs).

Expand Down Expand Up @@ -1220,31 +1220,27 @@ These are diagnostics the engine reports but cfn-lint does not expect (potential
- **F3016** `NoValuePoliciesWithoutTransform` (AWS::SQS::Queue) → `DeletionPolicy` L64 in `bad_lifecycle_policy_shapes_yaml`
> DeletionPolicy must be one of Delete, Retain, RetainExceptOnCreate, got null

### E3510 - 3 extra - Validate identity based IAM polices
### E3029 - 2 extra - Validate Route53 record set aliases

- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'NS'
- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'SOA'

### E3510 - 2 extra - Validate identity based IAM polices

- **E3510** `rIamPolicy` (AWS::IAM::Policy) → `Properties.PolicyDocument` L38 in `bad_resources_iam_iam_policy_yaml`
> [{"Statement":{}}] is not of type 'object'
- **E3510** `PolicyBadIdAndCondition` (AWS::IAM::RolePolicy) → `Properties.PolicyDocument.Id` L47 in `bad_resources_iam_identity_policy_e3510_yaml`
> Additional properties are not allowed ('Id' was unexpected)
- **E3510** `WildcardServicePolicy` (AWS::IAM::ManagedPolicy) → `Properties.PolicyDocument.Statement.0.Resource` L13 in `bad_resources_iam_identity_policy_wildcard_service_yaml`
> 'arn:aws:*:::example-bucket/*' does not match '^(arn:(aws[A-Za-z\-]*?|[A-Za-z?*\-]*[?*][A-Za-z?*\-]*):[^:*?]+:[^:]*(:(?:\d{12}|\*|aws)?:.+|)|\*)$'

### F3017 - 3 extra - Check Properties that need at least one of a list of properties
### F3017 - 2 extra - Check Properties that need at least one of a list of properties

- **F3017** `Topic` (AWS::SNS::Topic) → `Properties.KmsMasterKeyId` L11 in `bad_hardcoded_partition_yaml`
> Value is not valid under any of the 3 anyOf schemas (0 branches matched; at least one is required). Branch failures: branch 1: 'arn:aws:kms:us-east-1:123456789012:key/12345' does not match format 'AWS
- **F3017** `JoinedUsername` (AWS::RDS::DBCluster) → `Properties.MasterUsername` L9 in `bad_resources_rds_not_enum_master_username_join_yaml`
> 'rdsadmin' at 'MasterUsername' does not satisfy the composition branch constraint (none of ['rdsadmin']): 'rdsadmin' must not be one of ['rdsadmin']
- **F3017** `MyDB` (AWS::RDS::DBCluster) → `Properties.MasterUsername` L7 in `bad_resources_rds_not_enum_master_username_yaml`
> 'rdsadmin' at 'MasterUsername' does not satisfy the composition branch constraint (none of ['rdsadmin']): 'rdsadmin' must not be one of ['rdsadmin']

### E3029 - 2 extra - Validate Route53 record set aliases

- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'NS'
- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'SOA'

### F3033 - 2 extra - Check if a string has between min and max number of values specified

- **F3033** `myRepository` (AWS::CodeCommit::Repository) → `Properties.RepositoryName` L7 in `bad_resources_properties_string_size_yaml`
Expand Down Expand Up @@ -17914,7 +17910,7 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- **W9054** `CertAuth` (AWS::ACMPCA::CertificateAuthorityActivation) → `Properties.Certificate` L8 in `bad_schema_write_only_yaml`
> Write-only property 'Certificate' of 'CertAuth' is referenced in output 'WriteOnlyOutput'

## Per-Template Breakdown - 171 templates with mismatches
## Per-Template Breakdown - 170 templates with mismatches

### `good_lifecycle_intrinsic_scenarios_yaml` - 16 mismatches (0 TP, 0 FP, 0 EE, 16 FN)

Expand Down Expand Up @@ -18054,12 +18050,6 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FN: `F2015` ×4
- EE: `F2012` ×8

### `bad_resources_iam_iam_policy_yaml` - 4 mismatches (20 TP, 1 FP, 4 EE, 3 FN)

- FN: `F3003` ×3
- FP: `E3510`
- EE: `E1028`, `I9001`, `I9040`, `W2512`

### `bad_resources_iam_resource_policy_yaml` - 4 mismatches (0 TP, 0 FP, 2 EE, 4 FN)

- FN: `E3513` ×4
Expand Down Expand Up @@ -18135,6 +18125,11 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FP: `E3639` ×3
- EE: `I9001` ×10, `I9040` ×7, `F3003`

### `bad_resources_iam_iam_policy_yaml` - 3 mismatches (20 TP, 0 FP, 4 EE, 3 FN)

- FN: `F3003` ×3
- EE: `E1028`, `I9001`, `I9040`, `W2512`

### `bad_resources_properties_string_size_yaml` - 3 mismatches (3 TP, 2 FP, 3 EE, 1 FN)

- FN: `E3065`
Expand Down Expand Up @@ -18473,11 +18468,6 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FN: `E1017`
- EE: `I9001` ×8, `I9040` ×4

### `bad_hardcoded_partition_yaml` - 1 mismatches (0 TP, 1 FP, 5 EE, 0 FN)

- FP: `F3017`
- EE: `I9001` ×2, `I9040` ×2, `W9013`

### `bad_invalid_mapping_structure_yaml` - 1 mismatches (1 TP, 0 FP, 2 EE, 1 FN)

- FN: `E7001`
Expand Down Expand Up @@ -18797,9 +18787,9 @@ other tool's output. They are excluded from precision/recall scoring.

| Cause | Count | % of FP | Rules |
|-------|------:|--------:|-------|
| Other | 48 | 53.93% | E0001, F0018, F2002, F3016, F3017, F3033 |
| Over-reporting property/intrinsic errors | 29 | 32.58% | E1155, E3001, E3019, E3022, E3029, E3055, E3510, E3639 |
| Stricter than cfn-lint (informational) | 12 | 13.48% | I3011 |
| Other | 47 | 54.02% | E0001, F0018, F2002, F3016, F3017, F3033 |
| Over-reporting property/intrinsic errors | 28 | 32.18% | E1155, E3001, E3019, E3022, E3029, E3055, E3510, E3639 |
| Stricter than cfn-lint (informational) | 12 | 13.79% | I3011 |

## Location Mismatches - 9 matched pairs disagree on line

Expand Down
62 changes: 26 additions & 36 deletions scripts/snapshots/report_rego_detailed.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# cloudformation-validate vs cfn-lint - Parity Report

> Generated: 2026-09-02 18:43:01
> Generated: 2026-09-03 11:08:29
> Engine: **rego**
> Detail level: **detailed**
> Matching: `(rule_id, resource_id, path)` two-pass with `(rule_id, resource_id)` fallback + aliases
Expand All @@ -23,22 +23,22 @@
| Metric | Value |
|--------|------:|
| True Positives | 4053 |
| False Positives (engine bugs) | 89 |
| False Positives (engine bugs) | 87 |
| Engine Extra (correct, cfn-lint gap) | 8257 |
| False Negatives (engine misses) | 356 |
| Precision | 97.85% |
| Precision | 97.90% |
| Recall | 91.93% |
| F1 | 94.80% |
| F1 | 94.82% |
| Unique rules detected | 238 |
| Perfect templates | 494/665 |
| Perfect templates | 495/665 |
| Location mismatches (matched pairs) | 9 |

### By Severity

| Severity | TP | FP | EE | FN | Precision | Recall |
|----------|---:|---:|---:|---:|----------:|-------:|
| Fatal | 447 | 14 | 81 | 145 | 96.96% | 75.51% |
| Error | 856 | 63 | 12 | 140 | 93.14% | 85.94% |
| Fatal | 447 | 13 | 81 | 145 | 97.17% | 75.51% |
| Error | 856 | 62 | 12 | 140 | 93.25% | 85.94% |
| Warning | 2080 | 0 | 371 | 61 | 100.00% | 97.15% |
| Info | 670 | 12 | 7793 | 10 | 98.24% | 98.53% |

Expand Down Expand Up @@ -1040,7 +1040,7 @@ but found another document
- **W6001** → `Outputs.ImportedValue.Value.Fn::ImportValue` L39 in `good_output_value_string_yaml`
> The output value {'Fn::ImportValue': 'SomeExportedName'} is an import from another output

## False Positives - 89 extra findings across 15 rules
## False Positives - 87 extra findings across 15 rules

These are diagnostics the engine reports but cfn-lint does not expect (potential bugs).

Expand Down Expand Up @@ -1220,31 +1220,27 @@ These are diagnostics the engine reports but cfn-lint does not expect (potential
- **F3016** `NoValuePoliciesWithoutTransform` (AWS::SQS::Queue) → `DeletionPolicy` L64 in `bad_lifecycle_policy_shapes_yaml`
> DeletionPolicy must be one of Delete, Retain, RetainExceptOnCreate, got null

### E3510 - 3 extra - Validate identity based IAM polices
### E3029 - 2 extra - Validate Route53 record set aliases

- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'NS'
- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'SOA'

### E3510 - 2 extra - Validate identity based IAM polices

- **E3510** `rIamPolicy` (AWS::IAM::Policy) → `Properties.PolicyDocument` L38 in `bad_resources_iam_iam_policy_yaml`
> [{"Statement":{}}] is not of type 'object'
- **E3510** `PolicyBadIdAndCondition` (AWS::IAM::RolePolicy) → `Properties.PolicyDocument.Id` L47 in `bad_resources_iam_identity_policy_e3510_yaml`
> Additional properties are not allowed ('Id' was unexpected)
- **E3510** `WildcardServicePolicy` (AWS::IAM::ManagedPolicy) → `Properties.PolicyDocument.Statement.0.Resource` L13 in `bad_resources_iam_identity_policy_wildcard_service_yaml`
> 'arn:aws:*:::example-bucket/*' does not match '^(arn:(aws[A-Za-z\-]*?|[A-Za-z?*\-]*[?*][A-Za-z?*\-]*):[^:*?]+:[^:]*(:(?:\d{12}|\*|aws)?:.+|)|\*)$'

### F3017 - 3 extra - Check Properties that need at least one of a list of properties
### F3017 - 2 extra - Check Properties that need at least one of a list of properties

- **F3017** `Topic` (AWS::SNS::Topic) → `Properties.KmsMasterKeyId` L11 in `bad_hardcoded_partition_yaml`
> Value is not valid under any of the 3 anyOf schemas (0 branches matched; at least one is required). Branch failures: branch 1: 'arn:aws:kms:us-east-1:123456789012:key/12345' does not match format 'AWS
- **F3017** `JoinedUsername` (AWS::RDS::DBCluster) → `Properties.MasterUsername` L9 in `bad_resources_rds_not_enum_master_username_join_yaml`
> 'rdsadmin' at 'MasterUsername' does not satisfy the composition branch constraint (none of ['rdsadmin']): 'rdsadmin' must not be one of ['rdsadmin']
- **F3017** `MyDB` (AWS::RDS::DBCluster) → `Properties.MasterUsername` L7 in `bad_resources_rds_not_enum_master_username_yaml`
> 'rdsadmin' at 'MasterUsername' does not satisfy the composition branch constraint (none of ['rdsadmin']): 'rdsadmin' must not be one of ['rdsadmin']

### E3029 - 2 extra - Validate Route53 record set aliases

- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'NS'
- **E3029** `ConditionalInvalidAliasTypes` (AWS::Route53::RecordSet) → `Properties.AliasTarget` L48 in `bad_route53_conditional_scenarios_yaml`
> AliasTarget cannot be used with record type 'SOA'

### F3033 - 2 extra - Check if a string has between min and max number of values specified

- **F3033** `myRepository` (AWS::CodeCommit::Repository) → `Properties.RepositoryName` L7 in `bad_resources_properties_string_size_yaml`
Expand Down Expand Up @@ -17914,7 +17910,7 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- **W9054** `CertAuth` (AWS::ACMPCA::CertificateAuthorityActivation) → `Properties.Certificate` L8 in `bad_schema_write_only_yaml`
> Write-only property 'Certificate' of 'CertAuth' is referenced in output 'WriteOnlyOutput'

## Per-Template Breakdown - 171 templates with mismatches
## Per-Template Breakdown - 170 templates with mismatches

### `good_lifecycle_intrinsic_scenarios_yaml` - 16 mismatches (0 TP, 0 FP, 0 EE, 16 FN)

Expand Down Expand Up @@ -18054,12 +18050,6 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FN: `F2015` ×4
- EE: `F2012` ×8

### `bad_resources_iam_iam_policy_yaml` - 4 mismatches (20 TP, 1 FP, 4 EE, 3 FN)

- FN: `F3003` ×3
- FP: `E3510`
- EE: `E1028`, `I9001`, `I9040`, `W2512`

### `bad_resources_iam_resource_policy_yaml` - 4 mismatches (0 TP, 0 FP, 2 EE, 4 FN)

- FN: `E3513` ×4
Expand Down Expand Up @@ -18135,6 +18125,11 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FP: `E3639` ×3
- EE: `I9001` ×10, `I9040` ×7, `F3003`

### `bad_resources_iam_iam_policy_yaml` - 3 mismatches (20 TP, 0 FP, 4 EE, 3 FN)

- FN: `F3003` ×3
- EE: `E1028`, `I9001`, `I9040`, `W2512`

### `bad_resources_properties_string_size_yaml` - 3 mismatches (3 TP, 2 FP, 3 EE, 1 FN)

- FN: `E3065`
Expand Down Expand Up @@ -18473,11 +18468,6 @@ These are correct diagnostics the engine reports that cfn-lint does not cover.
- FN: `E1017`
- EE: `I9001` ×8, `I9040` ×4

### `bad_hardcoded_partition_yaml` - 1 mismatches (0 TP, 1 FP, 5 EE, 0 FN)

- FP: `F3017`
- EE: `I9001` ×2, `I9040` ×2, `W9013`

### `bad_invalid_mapping_structure_yaml` - 1 mismatches (1 TP, 0 FP, 2 EE, 1 FN)

- FN: `E7001`
Expand Down Expand Up @@ -18797,9 +18787,9 @@ other tool's output. They are excluded from precision/recall scoring.

| Cause | Count | % of FP | Rules |
|-------|------:|--------:|-------|
| Other | 48 | 53.93% | E0001, F0018, F2002, F3016, F3017, F3033 |
| Over-reporting property/intrinsic errors | 29 | 32.58% | E1155, E3001, E3019, E3022, E3029, E3055, E3510, E3639 |
| Stricter than cfn-lint (informational) | 12 | 13.48% | I3011 |
| Other | 47 | 54.02% | E0001, F0018, F2002, F3016, F3017, F3033 |
| Over-reporting property/intrinsic errors | 28 | 32.18% | E1155, E3001, E3019, E3022, E3029, E3055, E3510, E3639 |
| Stricter than cfn-lint (informational) | 12 | 13.79% | I3011 |

## Location Mismatches - 9 matched pairs disagree on line

Expand Down
22 changes: 12 additions & 10 deletions scripts/snapshots/rule_categorization_audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ severity model documented in `product.md`.

## Summary

- Total rules: **302**
- By severity: Fatal=69, Error=148, Warn=63, Info=22
- By true origin: CfnLint=203, Engine=29, Engine(collision)=1, Schema=69
- By category: BestPractice=49, Deprecation=8, Intrinsic=52, Parameter=8, Reference=2, Resource=87, Schema=25, Security=14, Structure=57
- cfn-lint reference: 322 rule IDs loaded
- Total rules: **303**
- By severity: Fatal=69, Error=148, Warn=64, Info=22
- By true origin: CfnLint=204, Engine=29, Engine(collision)=1, Schema=69
- By category: BestPractice=50, Deprecation=8, Intrinsic=52, Parameter=8, Reference=2, Resource=87, Schema=25, Security=14, Structure=57
- cfn-lint reference: 323 rule IDs loaded
- E→F promoted rules: 51
- Engine-extra rules: 60

Expand All @@ -29,10 +29,11 @@ True origin is computed by checking cfn-lint source, not the registry's
For non-Fatal CfnLint-origin rules, our description should align with
cfn-lint's `shortdesc`. Fatal rules are exempt

### Hard mismatches (3) - likely different rule
### Hard mismatches (4) - likely different rule

| ID | Sev | Sim | Our description | cfn-lint shortdesc |
|----|-----|----:|------------------|--------------------|
| `E3001` | Error | 0.00 | Basic CloudFormation Resource Check | SAM resource-level properties are ignored |
| `E1022` | Error | 0.09 | Fn::Join requires a string delimiter and a list of strings or string-producing intrinsics | Join validation of parameters |
| `E1024` | Error | 0.09 | Fn::Cidr requires a CIDR-format ipBlock string and integer count/cidrBits | Cidr validation of parameters |
| `E1031` | Error | 0.09 | Fn::ToJsonString argument must be a non-empty array/object or a supported function | ToJsonString validation of parameters |
Expand Down Expand Up @@ -122,10 +123,10 @@ schema-validator extensions or Fatal schema rules).
| `E3721` | Error | Validate ReplicaMode value for Oracle and Db2 engines | DbInstanceReplicaMode.py |
| `E3722` | Error | Cookies Forward must be a valid enum when CachePolicyId is absent | DistributionCacheBehaviorForwardEnum.py |
| `E3723` | Error | ResourcePath must start with / when method settings are configured | StageMethodSettingsResourcePath.py |
| `E3724` | Error | Validate Globals section | GlobalsTransform.py |
| `E3724` | Error | Validate Globals section and IgnoreGlobals entries | GlobalsTransform.py |
| `W1034` | Warn | Validate the values that come from a Fn::FindInMap function | FindInMapResolved.py |
| `W1036` | Warn | Validate the values that come from a Fn::GetAZs function | GetAzResolved.py |
| `W2002` | Warn | Parameter type is not officially supported by CloudFormation | UnsupportedParameterType.py |
| `W3001` | Warn | SAM resource-level properties are ignored | Configuration.py |
| `W3699` | Warn | ReplicaMode is ignored for non-Oracle/Db2 engines | DbInstanceReplicaModeIgnored.py |
| `W3700` | Warn | Non-standard Domain values are converted to vpc | EipDomain.py |
| `W3701` | Warn | SSM Parameter Name should not use /aws/ or /ssm/ prefix | ParameterNamePrefix.py |
Expand Down Expand Up @@ -243,7 +244,7 @@ Same rule ID emitted with semantically different messages.
**`E3048`** - registry: "Validate ECS Fargate tasks have required properties and values"

- Cluster 1 (4 sites): "Fargate requires NetworkMode to be specified as 'awsvpc'"
- Cluster 2 (1 sites): "Fargate Cpu value {} is not valid. Must be one of {}"
- Cluster 2 (1 sites): "Fargate Cpu value {} is not valid. Valid sizes are {} CPU units or {} vCPU."
- Cluster 3 (2 sites): "Fargate does not support PlacementConstraints"

**`E3671`** - registry: "Validate block device mapping configuration"
Expand Down Expand Up @@ -303,7 +304,7 @@ with `cargo test -p cfn-validate --test engine_parity` for ground truth.

**CEL only (2):** `E3023`, `E3510`

_Scanned 257 CEL sites (195 IDs), 279 Rego sites (193 IDs)._
_Scanned 259 CEL sites (196 IDs), 283 Rego sites (194 IDs)._

## Appendix: full rule inventory

Expand Down Expand Up @@ -561,6 +562,7 @@ _Scanned 257 CEL sites (195 IDs), 279 Rego sites (193 IDs)._
| `W1102` | Warn | BestPractice | Engine | Engine | Invalid intrinsic function usage |
| `W1103` | Warn | Intrinsic | Engine | Engine(collision) | Unknown intrinsic function name |
| `W2001` | Warn | BestPractice | CfnLint | CfnLint | Check if Parameters are Used |
| `W2002` | Warn | BestPractice | CfnLint | CfnLint | Parameter type is not officially supported by CloudFormation |
| `W2010` | Warn | Security | CfnLint | CfnLint | NoEcho parameters are not masked when used in Metadata and Outputs |
| `W2501` | Warn | Security | CfnLint | CfnLint | Check if Password Properties are correctly configured |
| `W2502` | Warn | BestPractice | Engine | Engine | DependsOn conditional resource without matching condition |
Expand Down
Loading
Loading