Merge branch 'main' into add/sarif_import_issue_with_ssl_troubleshooting

adrienpessu · web-flow · commit b539eec27404 · 2023-05-12T15:50:04.000+02:00
diff --git a/reporting/advanced-security-reporting.md b/reporting/advanced-security-reporting.md
@@ -18,4 +18,5 @@
   - [ ] https://github.com/ThibaudLopez/GHAS
 - SIEM integrations
   - [ ] https://github.blog/2022-10-13-introducing-github-advanced-security-siem-integrations-for-security-professionals/
+  - [ ] https://github.blog/2023-03-10-introducing-github-vulnerability-management-integrations-for-security-professionals/
   - [ ] https://resources.github.com/security/integrating-github-advanced-security-with-third-party-platforms/
diff --git a/reporting/issues_csv/README.md b/reporting/issues_csv/README.md
@@ -0,0 +1,2 @@
+A `powershell` script that fetches Code Scanning, Secret Scanning and Dependabot alerts for an organization and outputs them to a CSV file using `jq`.
+Includes the repository topics that might be used for filtering and grouping the alerts.
diff --git a/reporting/issues_csv/code_scanning.jq b/reporting/issues_csv/code_scanning.jq
@@ -0,0 +1,6 @@
+### the csv headers
+["repo","severity","created","fixed","dismissed","dismissed reason","state","url","topics"], 
+(.[]| 
+### the json path
+[.repository.name,.rule.severity,.created_at,.fixed_at,.dismissed_at,.dismissed_reason,.state,.html_url,($topics[][.repository.name]|join(" "))]
+) | @csv
diff --git a/reporting/issues_csv/dependabot.jq b/reporting/issues_csv/dependabot.jq
@@ -0,0 +1,16 @@
+### the csv headers
+["repo","package","severity","CVSS","created","fixed","dismissed","dismissed reason","state","url","topics"], 
+(.[].data.repository.vulnerabilityAlerts.edges[0].node | select(.!=null)|
+### the json path
+[.repository.name,
+ .securityVulnerability.package.name,
+ .securityVulnerability.severity,
+ .securityVulnerability.advisory.cvss.score,
+ .createdAt,
+ .fixedAt,
+ .dismissedAt,
+ .dismissReason,
+ .state,
+ ("https://github.com/beazley/"+.repository.name+"/security/dependabot/"+(.number|tostring)),
+ ($topics[][.repository.name]|join(" "))]
+)  | @csv
diff --git a/reporting/issues_csv/reporting.ps1 b/reporting/issues_csv/reporting.ps1
@@ -0,0 +1,26 @@
+#!/usr/bin/env pwsh
+$ORG = "mbaluda-org"
+
+### FETCH TOPICS ###
+$topics = gh api --cache 5m orgs/$ORG/repos -q 'map(select(.name)|{(.name):(.topics)})|add' | jq -s 'add'
+$topics | Out-File topics_map.json -encoding utf8
+
+### CODE SCANNING ALERTS ###
+gh api orgs/$ORG/code-scanning/alerts --method GET --paginate | jq -rf code_scanning.jq --slurpfile topics topics_map.json > code_scanning.csv
+ 
+### SECRET SCANNING ALERTS ###
+gh api orgs/$ORG/secret-scanning/alerts --method GET --paginate | jq -rf secret_scanning.jq --slurpfile topics topics_map.json > secret_scanning.csv
+ 
+### DEPENDABOT SCANNING ALERTS ###
+$repos = $topics | jq 'keys[]'
+$(foreach ($repo in $repos) {
+    gh api graphql -F group=$ORG -F repo=$repo -f query='
+  query ($endCursor: String, $group: String!, $repo: String!) {
+    repository(owner: $group, name: $repo) {
+      vulnerabilityAlerts(first: 100, after: $endCursor) {
+        edges { node { createdAt fixedAt dismissedAt dismissReason state securityVulnerability { package { name } severity advisory { cvss { score } } } repository { name } number } }
+        pageInfo { hasNextPage endCursor }
+      }
+    }
+  }' --paginate
+  }) | jq -srf dependabot.jq --slurpfile topics topics_map.json > dependabot.csv
diff --git a/reporting/issues_csv/secret_scanning.jq b/reporting/issues_csv/secret_scanning.jq
@@ -0,0 +1,6 @@
+### the csv headers
+["repo","type","created","fixed","resolution","state","url","topics"], 
+(.[] | select(.rule.severity!="severity")| 
+### the json path
+[.repository.name,.secret_type,.created_at,.resolved_at,.resolution,.state,.html_url,($topics[][.repository.name]|join(" "))]
+) | @csv
diff --git a/troubleshooting/codeql-builds/compiled-languages-csharp.md b/troubleshooting/codeql-builds/compiled-languages-csharp.md
@@ -55,7 +55,22 @@ Using `dotnet` is best documented at: https://docs.github.com/en/actions/automat
 #### NuGet Error NU1301
 This can indicate your custom package server is not configured which may fail the `dotnet restore` command.  For private package servers, the follwing guidance shows how to add package sources: [Setting up authentication for nuget feeds](https://github.com/actions/setup-dotnet#setting-up-authentication-for-nuget-feeds)
 
-### .NET Framework Manual Build Steps on Windows Runners
+#### NuGet.targets(132,5): warning : Your request could not be authenticated by the GitHub Packages service. Please ensure your access token is valid and has the appropriate scopes configured.
+
+Consider adding auth for your GitHub Packages hosted NuGet feed using the nuget CLI tooling.  Add this before the `autobuild` / custom build steps in your workflow.
+
+```yml 
+  - name: add nuget auth
+    run: dotnet nuget add source https://nuget.pkg.github.com/<org-goes-here>/index.json -n "GitHub" -u USERNAME -p "${{ secrets.GH_PACKAGES_READ_ONLY }}" --store-password-in-clear-text
+ ```
+
+
+### .NET Framework 
+
+#### NuGet Authentication
+Utilize the [nuget/setup-nuget](https://github.com/nuget/setup-nuget#basic) action to pass package key/source to nuget exe.
+
+#### Manual Build Steps on Windows Runners
 NOTE: if you require windows OS to build, ensure you are using a windows runner. 
 
 Example using `windows-latest`:
@@ -118,6 +133,21 @@ Running low on disk using the default Actions runner? Try a few of these workaro
 			
 - See also: [Vertical Scaling](#vertical-scaling---throw-hardware-at-the-software-problem)
 
+## MvcBuildViews target failures
+
+This can manifest through a variety of errors
+- `error ASPPARSE`
+- `[error]C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\web.config(113,0): Error ASPCONFIG: Could not load type`
+- `Error ASPCONFIG: It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level.`
+- `(AfterBuildCompiler target) -> D:\a\Orchard\Orchard\src\Orchard.Web\Modules\Orchard.Glimpse\web.config(38): error ASPCONFIG: Could not load file or assembly 'System.Web.Mvc, Version=5.2.3, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)`
+
+The CodeQL compiler tracer used for `csharp` will auto inject the /p:MvcBuildViews=true flag.  This pre-compilation of Views gives us the ability to extract the generated code from those files, leading to (potentially) better error reporting and location information if a query does flag an issue. The lack of view information passing through CodeQL to the compiler will lead to an incomplete database, where important dataflow sources/sinks/taint-steps are not included in the analysis.
+
+The recommendation here is to ensure that passing /p:MvcBuildViews=true to your CI build will compile even outside of CodeQL. Having a developer reivew this on their local machine is the best scenario. This can be on done on the specific web project by adding `<MvcBuildViews>true</MvcBuildViews>` to the local .csproj ( you will often find this defaulted to false).   The MVC full framework steps are listed [here](https://learn.microsoft.com/en-us/archive/blogs/jimlamb/turn-on-compile-time-view-checking-for-asp-net-mvc-projects-in-tfs-build-2010). There are a few different reasons why this can cause your project to fail compilation.
+
+For `Error ASPCONFIG: It is an error to use a section registered as allowDefinition='MachineToApplication' beyond application level.`,  change the locations of the obj and publish folder to not be located under the project folder of the website.  If you have bin/obj files checked into source then this could be a likely culprit: https://gunnarpeipman.com/aspnet-mvc-allowdefinition-machinetoapplication/.  You will find [various permutations of this recommendation](https://stackoverflow.com/questions/12778088/allowdefinition-machinetoapplication-error-setting-mvcbuildviewstrue-mvcbui) out there! 
+
+For `Error ASPCONFIG: Could not load type 'X.Y.Z'`, ensure that you do not have excluded `.cshtml`, `.ashx`, `.ashx.cs`, `.aspx` or `.aspx.cs` files on disk in existing `Views` folders or the Root folder of your project!  You can show hidden files in your solution view to hunt these down and remove from these folders.  MvcBuildViews does not observe the file include from the csproj when compiling the application. You may have to hunt these down one by one, so adding `<MvcBuildViews>true</MvcBuildViews>` to your local .csproj may help you get this done on your local machine with Visual Studio.  The `Error List` view in Visual Studio will have a column that shows you the actual File name you need to delete.
 
 # Speed up C# Analysis
 
@@ -126,10 +156,10 @@ Start here: [CodeQL Docs -  The build takes too long](https://docs.github.com/en
 ## Optimization - Caching Dependencies
  Depending on the number of dependencies, it may be faster to restore packages for your project using the Actions dependency cache. Projects with many large dependencies should see a performance increase as it cuts down the time required for downloading. Projects with fewer dependencies may not see a significant performance increase and may even see a slight decrease due to how NuGet installs cached dependencies. The performance varies from project to project. See [this article](https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-net#caching-dependencies) for configuring the NuGet dependency cache.
 
-## Optimization - Removing Unit Tests
-CodeQL will extract and analyze any code that is passed through the compiler.  Consider excluding any code you do not wish to include in a security scan to speed up and remove noise from this process.
+## Optimization - Removing Code From Scans
+CodeQL will extract and analyze any code that is passed through the compiler.  Consider excluding any code you do not wish to include in a security scan to speed up and remove noise from this process. This is commonly employed for unit tests, demo code, or code that would not benefit from being scanned (ex: DacPacs).
 
-With .NET we can employ a few mechanisms to remove test/demo code from CodeQL scans (e.g. you would want to run your unit test in another workflow ):
+With .NET we can employ a few mechanisms to remove code from CodeQL scans (e.g. you would want to run your unit test in another workflow ):
 - A [solution filter](https://docs.microsoft.com/en-us/visualstudio/msbuild/solution-filters?view=vs-2019) to only build required projects
 - An explicit [solution file that excludes projects](https://docs.microsoft.com/en-us/visualstudio/ide/how-to-exclude-projects-from-a-build?view=vs-2022)
    - example from the Open Source project: [Identity Server](https://github.com/DuendeSoftware/IdentityServer/)
diff --git a/troubleshooting/codeql-builds/compiled-languages-java.md b/troubleshooting/codeql-builds/compiled-languages-java.md
@@ -1,10 +1,10 @@
 
 # Private Package Registries
 
-## The codeql for java is failing when it tries to do mvn command and tries to access a artifactory repo where our pom.xml are stored.
-
-Assuming the given package registry instance is publicly accessible:
+## The autobuild for java is failing when running Maven build command and a private package registry is needed - `status: 401 Unauthorized `
+- ex: artifactory where our pom.xml dependencies are stored
 
+Assuming the given package registry instance is publicly accessible and needs credentials:
 
 Option 1 - Pass credentials via environment variable from Actions secrets and configure Maven settings to utilize those credentials (see sample [here](https://github.com/actions/setup-java/blob/main/docs/advanced-usage.md#yaml-example))
 
@@ -24,7 +24,20 @@ ex `settings.xml`
     </server>
 ```
 
-Option 2 - Use the [maven-settings-action](https://github.com/s4u/maven-settings-action) to dynamically create/overrite a `settings.xml` that contains the credentials for your specified package manager.
+Option 2 - Use the GitHub https://github.com/actions/setup-java#maven-options action to generate maven's settings.xml on the fly and pass the values to Apache Maven GPG Plugin as well as Apache Maven Toolchains.
+
+```yml
+    - name: Set up Apache Maven Central
+      uses: actions/setup-java@v3
+      with: 
+        distribution: 'temurin'
+        java-version: '11'
+        server-id: maven # Value of the distributionManagement/repository/id field of the pom.xml
+        server-username: MAVEN_USERNAME # env variable for username in deploy
+        server-password: MAVEN_CENTRAL_TOKEN # env variable for token in deploy
+ ```
+
+Option 3 - Use the [maven-settings-action](https://github.com/s4u/maven-settings-action) to dynamically create/overrite a `settings.xml` that contains the credentials for your specified package manager.
 
 ```yml
 - if: matrix.language == 'java'
@@ -34,6 +47,8 @@ Option 2 - Use the [maven-settings-action](https://github.com/s4u/maven-settings
         servers: '[{"id": "central", "username": "${{ secrets.MAVEN_USERNAME }}", "password": "${{ secrets.MAVEN_CENTRAL_TOKEN }}"}]'
 ```
 
+See also: [401 due to private package server configuration](compiled-languages.md#401-due-to-private-package-server-configuration)
+
 # Build Failures
 
 ## java.lang.IllegalArgumentException: Unsupported class file major version ##
@@ -42,10 +57,15 @@ Ensure you are compiling your java application using CodeQL tracing on a support
 
 ## Fatal error compiling: error: invalid target release: \## 
 
-Specify your [desired java version via the setup-java action](https://github.com/actions/setup-java#supported-version-syntax)
+Alternative error:  
+```
+> error: invalid source release:
+```
+
+Resolution here is to specify your [desired java version via the setup-java action](https://github.com/actions/setup-java#supported-version-syntax)
 ```yml
- uses: actions/setup-java@v3
- with:
-     java-version: 17
-     distribution: 'microsoft'
-```
+- uses: actions/setup-java@v3
+  with:
+    java-version: 17
+    distribution: 'microsoft'
+```
diff --git a/troubleshooting/codeql-builds/compiled-languages.md b/troubleshooting/codeql-builds/compiled-languages.md
@@ -24,6 +24,7 @@ See [language specific guidance](#language-specific-guidance) for common resolut
 Ensure network access from GitHub runners to your private registry is open
    - For IP Whitelisting, consider using [Larger Runners with Static IP](https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners#networking-for-larger-runners)
    - See Also: [Connecting Actions to a private network](https://docs.github.com/en/actions/using-github-hosted-runners/connecting-to-a-private-network)
+   - Alternatively, consider a self-hosted actions runner that will execute within your existing private network. See ["Hosting your own runners"](https://docs.github.com/en/actions/hosting-your-own-runners)
 
 See [language specific guidance](#language-specific-guidance) for authentication options to popular package mangers 
 
@@ -48,9 +49,9 @@ alternatively we can further define limits
     - name: Perform CodeQL Analysis
       uses: github/codeql-action/analyze@v2
       with: 
-	    # Increase Values seen in logs:
+        # Increase Values seen in logs:
         #2022-06-01T19:37:19.0200037Z   CODEQL_RAM: 119741
-	    #2022-06-01T19:37:19.0200307Z   CODEQL_THREADS: 32
+        #2022-06-01T19:37:19.0200307Z   CODEQL_THREADS: 32
         ram: 64000
         threads: 16
 ```
@@ -64,5 +65,6 @@ Helpful Articles to understand how to review, troubleshoot, and debug logs:
 - [Adding artifacts on every CodeQL Run](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/troubleshooting-the-codeql-workflow#creating-codeql-debugging-artifacts-using-a-workflow-flag)
 - [Exit Codes](https://codeql.github.com/docs/codeql-cli/exit-codes/)
 
-## Optimizaitons
-- CodeQL Docs -  [The build takes too long](https://docs.github.com/en/enterprise-cloud@latest/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/troubleshooting-the-codeql-workflow)
+
+## Optimizations
+- CodeQL Docs -  [The build takes too long](https://docs.github.com/en/enterprise-cloud@latest/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/troubleshooting-the-codeql-workflow)
diff --git a/troubleshooting/sarif-upload/troubleshooting.md b/troubleshooting/sarif-upload/troubleshooting.md
@@ -1,4 +1,5 @@
-### SARIF Upload Errors
+## SARIF Upload Errors
+* Test environment - GHES 3.2.1 + CodeQL CLI 2.7.2
 
 :gift: wrong ref:
 ```
@@ -58,9 +59,37 @@ More information on the API can be found [here](https://docs.github.com/en/rest/
 
 ### Test environments
 - GHES 3.2.1 + CodeQL CLI 2.7.2
+=======
+## SARIF Parsing Errors
 
+### Code Scanning could not process the submitted SARIF file: rejecting SARIF, as there are more runs than allowed (123 > 15)
+The GitHub api for accepting SARIF uploads has a limiter to prevent that number from being greater than specified (>15) for each upload.
 
-### Tools to rewrite SARIF
+See limits for various thresholds on the [REST API documentation](https://docs.github.com/en/rest/code-scanning?apiVersion=2022-11-28#upload-an-analysis-as-sarif-data)
+* Runs per file
+* Results per run
+* Rules per run	
+* Tool extensions per run
+* Thread Flow Locations per result
+* Location per result
+* Tags per rule	
+
+### A fatal error occurred: SARIF file is too large. The GitHub code scanning API accepts a max file size of 2000MB. This file is xxxxMB. File: "xyz.sarif"
+- aleternatively - `failed decompressing file from the path: "upload /xyz.sarif.gz": maximum SARIF size exceeded`
+
+First, review recommendedations per language to reduce the amount of code being scanned (e.g. removing test or demo code from the scan in an attempt to remove unwanted detections from SARIF). A detailed analysis of the SARIF file may indicate a massive number of a single rule, in this case [excluding a specific rule from the analysis](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/customizing-code-scanning#excluding-specific-queries-from-analysis) would be the best solution.  Alternatively, use a tool like [filter-sarif action](https://github.com/advanced-security/filter-sarif) to rewrite the SARIF file to exclude specific detections via an exclusion pattern. 
+
+If there are many deep code paths highlighted in the SARIF, use `--max-path=0` (or 1) passed into the analyze step or `database analyze` cli command to get rid of the dataflow paths and reduce the SARIF size that way (NOTE this will impact all rules). 
+
+```yml
+- name: Perform CodeQL Analysis
+  uses: github/codeql-action/analyze@v2
+  env: 
+     CODEQL_ACTION_EXTRA_OPTIONS: '{"database":{"interpret-results":["--max-paths", 1]}}'
+```
+
+## Tools to rewrite SARIF
 - `jq`
 - [Microsoft's SARIF tool](https://github.com/microsoft/sarif-sdk/blob/main/docs/multitool-usage.md)
 - [Dr. House's SARIF CLI](https://github.com/hohn/sarif-cli)
+- [advanced-security/filter-sarif action](https://github.com/advanced-security/filter-sarif)

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	+A `powershell` script that fetches Code Scanning, Secret Scanning and Dependabot alerts for an organization and outputs them to a CSV file using `jq`.
	`2`	`+Includes the repository topics that might be used for filtering and grouping the alerts.`