add endpoint custom documentation to generate-docs #389

ferullo · 2023-07-17T16:12:42Z

Change Summary

This is a replacement PR for #334

This PR is the first of four chained PRs. I've broken them up to make them reviewable. The final goal is to document what fields are actually available in each Endpoint document type.

(This PR) Add the code to read in new YAML documentation and render it in Markdown.
Add minimal YAML documentation that defines all the document types add minimal custom_documentation files #390
Update all YAML documentation to add content to custom_documentation files #391
Render the YAML into pre-committed Markdown files add rendered custom_documentation #392

Please at least skim all four PRs because seeing them all will help understand this PR better. Notably, I have comments asking for feedback in the second PR that are relevant to this PR.

Release Target

This will ideally be backported into 8.9

ferullo · 2023-07-17T16:13:18Z

scripts/generate-docs/main.go

@@ -67,6 +69,11 @@ func generateDocs(options generateOptions) error {
 	}

 	for _, packageName := range packages {
+		err = renderCustomDocumentation(options, packageName)
+		if err != nil {
+			return errors.Wrapf(err, "rendering custom documentation failed (packageName: %s)", packageName)


Does anyone have any ideas for a better conceptual name for this documentation than "custom documentation"?

"Detailed Documentation" ?

enumerated docs?

packeted docs? (or some other word to logically describe the grouping breakdown of the payload types. By OS / feature)

elasticmachine · 2023-07-17T16:16:09Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-28T02:28:53.682+0000
Duration: 7 min 23 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

ferullo · 2023-07-17T16:25:47Z

scripts/generate-docs/custom_doc.go

+			for _, f := range event.doc.Fields.Endpoint {
+				detail, ok := event.doc.Fields.Details[f]
+				if ok {
+					f += "<br /><br />" + strings.TrimSpace(strings.ReplaceAll(detail.Description, "\n", " "))


here is a relevant comment in another PRs that shows how this works.

https://github.com/elastic/endpoint-package/pull/391/files#r1265607445

ferullo · 2023-07-17T16:27:43Z

scripts/generate-docs/custom_doc.go

+}
+
+type eventFields struct {
+	Endpoint []string                     `yaml:"endpoint"`


I don't love this YAML key name (see comment in another PR here https://github.com/elastic/endpoint-package/pull/390/files#r1265610303)

ferullo · 2023-07-17T16:31:18Z

@kevinlog can this PR's general approach be reviewed. If it looks good an LGTM would be nice. Without any requests for changes in this or any other PRs in this chain the only change I plan to make is to fill in all the "FILL ME IN" placeholders.

kevinlog · 2023-07-17T16:57:27Z

Seeing failed make with the following message:

[2023-07-17T16:16:01.804Z] 2023/07/17 16:16:01 generating docs failed: rendering custom documentation failed (packageName: endpoint): failed to find custom documentation (path: custom_documentation/endpoint: Failed to list directory custom_documentation/endpoint/data_stream: open custom_documentation/endpoint/data_stream: no such file or directory

Jenkins: https://fleet-ci.elastic.co//blue/rest/organizations/jenkins/pipelines/endpoint-package-mbp/pipelines/PR-389/runs/1/steps/116/log/?start=0

From the read out, it looks like the custom_documentation is expecting there to be a data_stream of the same name?

ferullo · 2023-07-17T20:56:10Z

The make failure is an artifact of me breaking this into four PRs. The final PR in the chain (#392) passes the same checks.

I plan to merge PR4 into PR3, then PR3 into PR2, then PR2 into this PR, then this PR into main when it comes time to merge. I have no problem with only receiving comments that say LGTM before that merge flow happens if you'd like. But once I do it it'll be a lot harder to review just these changes.

pzl · 2023-07-18T13:06:45Z

scripts/generate-docs/main.go

@@ -67,6 +69,11 @@ func generateDocs(options generateOptions) error {
 	}

 	for _, packageName := range packages {
+		err = renderCustomDocumentation(options, packageName)
+		if err != nil {
+			return errors.Wrapf(err, "rendering custom documentation failed (packageName: %s)", packageName)


enumerated docs?

packeted docs? (or some other word to logically describe the grouping breakdown of the payload types. By OS / feature)

jdu2600 · 2023-07-20T03:39:27Z

doc_templates/endpoint/docs/CustomDocumentationREADME.md

+
+**This documentation is still beta**
+
+The subdirectories document all ECS fields that may exist in documents generated by Endpoint into


'document fields in documents' - phrasing is awkward

doc_templates/endpoint/docs/CustomDocumentationREADME.md

jdu2600 · 2023-07-20T03:44:27Z

scripts/generate-docs/custom_doc.go

+		low := strings.ToLower(_os)
+		if low != "linux" && low != "macos" && low != "windows" {


Why not enforce strict casing for consistency?
(And does go have a cleaner "item in list" syntax?)

Suggested change

low := strings.ToLower(_os)

if low != "linux" && low != "macos" && low != "windows" {

if _os != "linux" && _os != "macos" && _os != "windows" {

Ah. I see that you do enforce consistency later - on output.

jdu2600

LGTM - noting that I don't know enough go to know if this is good go

Co-authored-by: John U <[email protected]>

scripts/generate-docs/main.go

Co-authored-by: Leszek Kubik <[email protected]>

* add minimal custom_documentation files * fill in FILL ME IN * add linux and macos unquarantine * change heartbeat data stream type * yam => yaml * only credential access api events are non-diagnostic * Apply suggestions from code review Co-authored-by: Leszek Kubik <[email protected]> * add content to custom_documentation files (#391) * add content to custom_documentation files * add macos endpoint_unquarantine fields * linux endpoint unquarantine documentation * add more malicous behavior fields on linux * add more missing fields * add a field * switch to .* notation for malicious alert fields * restore host.* fields * add rendered custom_documentation (#392) * add rendered custom_documentation * re-render docs * re-render --------- Co-authored-by: Leszek Kubik <[email protected]>

ferullo · 2023-07-28T13:14:59Z

I merged the 3 chained PRs into this. There were minor updates to firm up some of the fields enumerated. I'll merge this PR into main in a few hours unless I hear any objections.

elasticmachine · 2023-08-14T18:33:16Z

Package endpoint - 8.10.0 containing this change is available at https://epr.elastic.co/search?package=endpoint

add endpoint custom documentation to generate-docs

8d09ab9

ferullo requested a review from a team as a code owner July 17, 2023 16:12

ferullo requested review from ashokaditya and paul-tavares July 17, 2023 16:12

ferullo commented Jul 17, 2023

View reviewed changes

ferullo mentioned this pull request Jul 17, 2023

add minimal custom_documentation files #390

Merged

This was referenced Jul 17, 2023

add content to custom_documentation files #391

Merged

add rendered custom_documentation #392

Merged

ferullo commented Jul 17, 2023

View reviewed changes

ferullo mentioned this pull request Jul 17, 2023

4420 update doc generator #334

Closed

add some README content

43452c9

pzl approved these changes Jul 18, 2023

View reviewed changes

jdu2600 reviewed Jul 20, 2023

View reviewed changes

doc_templates/endpoint/docs/CustomDocumentationREADME.md Outdated Show resolved Hide resolved

jdu2600 reviewed Jul 20, 2023

View reviewed changes

doc_templates/endpoint/docs/CustomDocumentationREADME.md Outdated Show resolved Hide resolved

jdu2600 reviewed Jul 20, 2023

View reviewed changes

jdu2600 approved these changes Jul 20, 2023

View reviewed changes

ferullo and others added 2 commits July 20, 2023 21:00

Update doc_templates/endpoint/docs/CustomDocumentationREADME.md

2254fb9

Co-authored-by: John U <[email protected]>

Update doc_templates/endpoint/docs/CustomDocumentationREADME.md

138e9c6

Co-authored-by: John U <[email protected]>

intxgo reviewed Jul 24, 2023

View reviewed changes

scripts/generate-docs/main.go Outdated Show resolved Hide resolved

Update scripts/generate-docs/main.go

d064993

Co-authored-by: Leszek Kubik <[email protected]>

intxgo approved these changes Jul 25, 2023

View reviewed changes

paul-tavares removed their request for review July 26, 2023 13:51

ferullo and others added 2 commits July 27, 2023 19:09

normalize yaml format

cf1592d

ferullo merged commit 8f04708 into main Jul 31, 2023

ferullo deleted the endpoint-documentation-code branch July 31, 2023 16:34


		This documentation is still beta

		The subdirectories document all ECS fields that may exist in documents generated by Endpoint into

		low := strings.ToLower(_os)
		if low != "linux" && low != "macos" && low != "windows" {

	low := strings.ToLower(_os)
	if low != "linux" && low != "macos" && low != "windows" {
	if _os != "linux" && _os != "macos" && _os != "windows" {

add endpoint custom documentation to generate-docs #389

add endpoint custom documentation to generate-docs #389

Uh oh!

Conversation

ferullo commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Change Summary

Release Target

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ferullo commented Jul 17, 2023

Uh oh!

kevinlog commented Jul 17, 2023

Uh oh!

ferullo commented Jul 17, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jdu2600 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ferullo commented Jul 28, 2023

Uh oh!

elasticmachine commented Aug 14, 2023

Uh oh!

Uh oh!

ferullo commented Jul 17, 2023 •

edited

Loading

elasticmachine commented Jul 17, 2023 •

edited

Loading