THIS IS A TEST INSTANCE ONLY! REPOSITORIES CAN BE DELETED AT ANY TIME!

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

497 lines
20 KiB

Add resource auto grouping Sorry for the size of this patch, I was busy hacking and plumbing away and it got out of hand! I'm allowing this because there doesn't seem to be anyone hacking away on parts of the code that this would break, since the resource code is fairly stable in this change. In particular, it revisits and refreshes some areas of the code that didn't see anything new or innovative since the project first started. I've gotten rid of a lot of cruft, and in particular cleaned up some things that I didn't know how to do better before! Here's hoping I'll continue to learn and have more to improve upon in the future! (Well let's not hope _too_ hard though!) The logical goal of this patch was to make logical grouping of resources possible. For example, it might be more efficient to group three package installations into a single transaction, instead of having to run three separate transactions. This is because a package installation typically has an initial one-time per run cost which shouldn't need to be repeated. Another future goal would be to group file resources sharing a common base path under a common recursive fanotify watcher. Since this depends on fanotify capabilities first, this hasn't been implemented yet, but could be a useful method of reducing the number of separate watches needed, since there is a finite limit. It's worth mentioning that grouping resources typically _reduces_ the parallel execution capability of a particular graph, but depending on the cost/benefit tradeoff, this might be preferential. I'd submit it's almost universally beneficial for pkg resources. This monster patch includes: * the autogroup feature * the grouping interface * a placeholder algorithm * an extensive test case infrastructure to test grouping algorithms * a move of some base resource methods into pgraph refactoring * some config/compile clean ups to remove code duplication * b64 encoding/decoding improvements * a rename of the yaml "res" entries to "kind" (more logical) * some docs * small fixes * and more!
5 years ago
Add resource auto grouping Sorry for the size of this patch, I was busy hacking and plumbing away and it got out of hand! I'm allowing this because there doesn't seem to be anyone hacking away on parts of the code that this would break, since the resource code is fairly stable in this change. In particular, it revisits and refreshes some areas of the code that didn't see anything new or innovative since the project first started. I've gotten rid of a lot of cruft, and in particular cleaned up some things that I didn't know how to do better before! Here's hoping I'll continue to learn and have more to improve upon in the future! (Well let's not hope _too_ hard though!) The logical goal of this patch was to make logical grouping of resources possible. For example, it might be more efficient to group three package installations into a single transaction, instead of having to run three separate transactions. This is because a package installation typically has an initial one-time per run cost which shouldn't need to be repeated. Another future goal would be to group file resources sharing a common base path under a common recursive fanotify watcher. Since this depends on fanotify capabilities first, this hasn't been implemented yet, but could be a useful method of reducing the number of separate watches needed, since there is a finite limit. It's worth mentioning that grouping resources typically _reduces_ the parallel execution capability of a particular graph, but depending on the cost/benefit tradeoff, this might be preferential. I'd submit it's almost universally beneficial for pkg resources. This monster patch includes: * the autogroup feature * the grouping interface * a placeholder algorithm * an extensive test case infrastructure to test grouping algorithms * a move of some base resource methods into pgraph refactoring * some config/compile clean ups to remove code duplication * b64 encoding/decoding improvements * a rename of the yaml "res" entries to "kind" (more logical) * some docs * small fixes * and more!
5 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
lang: Add module imports and more This enables imports in mcl code, and is one of last remaining blockers to using mgmt. Now we can start writing standalone modules, and adding standard library functions as needed. There's still lots to do, but this was a big missing piece. It was much harder to get right than I had expected, but I think it's solid! This unfortunately large commit is the result of some wild hacking I've been doing for the past little while. It's the result of a rebase that broke many "wip" commits that tracked my private progress, into something that's not gratuitously messy for our git logs. Since this was a learning and discovery process for me, I've "erased" the confusing git history that wouldn't have helped. I'm happy to discuss the dead-ends, and a small portion of that code was even left in for possible future use. This patch includes: * A change to the cli interface: You now specify the front-end explicitly, instead of leaving it up to the front-end to decide when to "activate". For example, instead of: mgmt run --lang code.mcl we now do: mgmt run lang --lang code.mcl We might rename the --lang flag in the future to avoid the awkward word repetition. Suggestions welcome, but I'm considering "input". One side-effect of this change, is that flags which are "engine" specific now must be specified with "run" before the front-end name. Eg: mgmt run --tmp-prefix lang --lang code.mcl instead of putting --tmp-prefix at the end. We also changed the GAPI slightly, but I've patched all code that used it. This also makes things consistent with the "deploy" command. * The deploys are more robust and let you deploy after a run This has been vastly improved and let's mgmt really run as a smart engine that can handle different workloads. If you don't want to deploy when you've started with `run` or if one comes in, you can use the --no-watch-deploy option to block new deploys. * The import statement exists and works! We now have a working `import` statement. Read the docs, and try it out. I think it's quite elegant how it fits in with `SetScope`. Have a look. As a result, we now have some built-in functions available in modules. This also adds the metadata.yaml entry-point for all modules. Have a look at the examples or the tests. The bulk of the patch is to support this. * Improved lang input parsing code: I re-wrote the parsing that determined what ran when we passed different things to --lang. Deciding between running an mcl file or raw code is now handled in a more intelligent, and re-usable way. See the inputs.go file if you want to have a look. One casualty is that you can't stream code from stdin *directly* to the front-end, it's encapsulated into a deploy first. You can still use stdin though! I doubt anyone will notice this change. * The scope was extended to include functions and classes: Go forth and import lovely code. All these exist in scopes now, and can be re-used! * Function calls actually use the scope now. Glad I got this sorted out. * There is import cycle detection for modules! Yes, this is another dag. I think that's #4. I guess they're useful. * A ton of tests and new test infra was added! This should make it much easier to add new tests that run mcl code. Have a look at TestAstFunc1 to see how to add more of these. As usual, I'll try to keep these commits smaller in the future!
3 years ago
  1. # General documentation
  2. ## Overview
  3. The `mgmt` tool is a next generation config management prototype. It's not yet
  4. ready for production, but we hope to get there soon. Get involved today!
  5. ## Project Description
  6. The mgmt tool is a distributed, event driven, config management tool, that
  7. supports parallel execution, and librarification to be used as the management
  8. foundation in and for, new and existing software.
  9. For more information, you may like to read some blog posts from the author:
  10. * [Next generation config mgmt](https://purpleidea.com/blog/2016/01/18/next-generation-configuration-mgmt/)
  11. * [Automatic edges in mgmt](https://purpleidea.com/blog/2016/03/14/automatic-edges-in-mgmt/)
  12. * [Automatic grouping in mgmt](https://purpleidea.com/blog/2016/03/30/automatic-grouping-in-mgmt/)
  13. * [Automatic clustering in mgmt](https://purpleidea.com/blog/2016/06/20/automatic-clustering-in-mgmt/)
  14. * [Remote execution in mgmt](https://purpleidea.com/blog/2016/10/07/remote-execution-in-mgmt/)
  15. * [Send/Recv in mgmt](https://purpleidea.com/blog/2016/12/07/sendrecv-in-mgmt/)
  16. * [Metaparameters in mgmt](https://purpleidea.com/blog/2017/03/01/metaparameters-in-mgmt/)
  17. There is also an [introductory video](https://www.youtube.com/watch?v=LkEtBVLfygE&html5=1)
  18. available. Older videos and other material [is available](on-the-web.md).
  19. ## Setup
  20. You'll probably want to read the [quick start guide](quick-start-guide.md) to
  21. get going.
  22. ## Features
  23. This section details the numerous features of mgmt and some caveats you might
  24. need to be aware of.
  25. ### Autoedges
  26. Automatic edges, or AutoEdges, is the mechanism in mgmt by which it will
  27. automatically create dependencies for you between resources. For example,
  28. since mgmt can discover which files are installed by a package it will
  29. automatically ensure that any file resource you declare that matches a
  30. file installed by your package resource will only be processed after the
  31. package is installed.
  32. #### Controlling autoedges
  33. Though autoedges is likely to be very helpful and avoid you having to declare
  34. all dependencies explicitly, there are cases where this behaviour is
  35. undesirable.
  36. Some distributions allow package installations to automatically start the
  37. service they ship. This can be problematic in the case of packages like MySQL
  38. as there are configuration options that need to be set before MySQL is ever
  39. started for the first time (or you'll need to wipe the data directory). In
  40. order to handle this situation you can disable autoedges per resource and
  41. explicitly declare that you want `my.cnf` to be written to disk before the
  42. installation of the `mysql-server` package.
  43. You can disable autoedges for a resource by setting the `autoedge` key on
  44. the meta attributes of that resource to `false`.
  45. #### Blog post
  46. You can read the introductory blog post about this topic here:
  47. [https://purpleidea.com/blog/2016/03/14/automatic-edges-in-mgmt/](https://purpleidea.com/blog/2016/03/14/automatic-edges-in-mgmt/)
  48. ### Autogrouping
  49. Automatic grouping or AutoGroup is the mechanism in mgmt by which it will
  50. automatically group multiple resource vertices into a single one. This is
  51. particularly useful for grouping multiple package resources into a single
  52. resource, since the multiple installations can happen together in a single
  53. transaction, which saves a lot of time because package resources typically have
  54. a large fixed cost to running (downloading and verifying the package repo) and
  55. if they are grouped they share this fixed cost. This grouping feature can be
  56. used for other use cases too.
  57. You can disable autogrouping for a resource by setting the `autogroup` key on
  58. the meta attributes of that resource to `false`.
  59. #### Blog post
  60. You can read the introductory blog post about this topic here:
  61. [https://purpleidea.com/blog/2016/03/30/automatic-grouping-in-mgmt/](https://purpleidea.com/blog/2016/03/30/automatic-grouping-in-mgmt/)
  62. ### Automatic clustering
  63. Automatic clustering is a feature by which mgmt automatically builds, scales,
  64. and manages the embedded etcd cluster which is compiled into mgmt itself. It is
  65. quite helpful for rapidly bootstrapping clusters and avoiding the extra work to
  66. setup etcd.
  67. If you prefer to avoid this feature. you can always opt to use an existing etcd
  68. cluster that is managed separately from mgmt by pointing your mgmt agents at it
  69. with the `--seeds` variable.
  70. #### Blog post
  71. You can read the introductory blog post about this topic here:
  72. [https://purpleidea.com/blog/2016/06/20/automatic-clustering-in-mgmt/](https://purpleidea.com/blog/2016/06/20/automatic-clustering-in-mgmt/)
  73. ### Remote ("agent-less") mode
  74. Remote mode is a special mode that lets you kick off mgmt runs on one or more
  75. remote machines which are only accessible via SSH. In this mode the initiating
  76. host connects over SSH, copies over the `mgmt` binary, opens an SSH tunnel, and
  77. runs the remote program while simultaneously passing the etcd traffic back
  78. through the tunnel so that the initiators etcd cluster can be used to exchange
  79. resource data.
  80. The interesting benefit of this architecture is that multiple hosts which can't
  81. connect directly use the initiator to pass the important traffic through to each
  82. other. Once the cluster has converged all the remote programs can shutdown
  83. leaving no residual agent.
  84. This mode can also be useful for bootstrapping a new host where you'd like to
  85. have the service run continuously and as part of an mgmt cluster normally.
  86. In particular, when combined with the `--converged-timeout` parameter, the
  87. entire set of running mgmt agents will need to all simultaneously converge for
  88. the group to exit. This is particularly useful for bootstrapping new clusters
  89. which need to exchange information that is only available at run time.
  90. This existed in earlier versions of mgmt as a `--remote` option, but it has been
  91. removed and is being ported to a more powerful variant where you can remote
  92. execute via a `remote` resource.
  93. #### Blog post
  94. You can read the introductory blog post about this topic here:
  95. [https://purpleidea.com/blog/2016/10/07/remote-execution-in-mgmt/](https://purpleidea.com/blog/2016/10/07/remote-execution-in-mgmt/)
  96. ### Puppet support
  97. You can supply a Puppet manifest instead of creating the (YAML) graph manually.
  98. Puppet must be installed and in `mgmt`'s search path. You also need the
  99. [ffrank-mgmtgraph Puppet module](https://forge.puppet.com/ffrank/mgmtgraph).
  100. Invoke `mgmt` with the `--puppet` switch, which supports 3 variants:
  101. 1. Request the configuration from the Puppet Master (like `puppet agent` does)
  102. `mgmt run puppet --puppet agent`
  103. 2. Compile a local manifest file (like `puppet apply`)
  104. `mgmt run puppet --puppet /path/to/my/manifest.pp`
  105. 3. Compile an ad hoc manifest from the commandline (like `puppet apply -e`)
  106. `mgmt run puppet --puppet 'file { "/etc/ntp.conf": ensure => file }'`
  107. For more details and caveats see [puppet-guide.md](puppet-guide.md).
  108. #### Blog post
  109. An introductory post on the Puppet support is on
  110. [Felix's blog](http://ffrank.github.io/features/2016/06/19/puppet-powered-mgmt/).
  111. ## Reference
  112. Please note that there are a number of undocumented options. For more
  113. information on these options, please view the source at:
  114. [https://github.com/purpleidea/mgmt/](https://github.com/purpleidea/mgmt/).
  115. If you feel that a well used option needs documenting here, please patch it!
  116. ### Overview of reference
  117. * [Meta parameters](#meta-parameters): List of available resource meta parameters.
  118. * [Lang metadata file](#lang-metadata-file): Lang metadata file format.
  119. * [Graph definition file](#graph-definition-file): Main graph definition file.
  120. * [Command line](#command-line): Command line parameters.
  121. * [Compilation options](#compilation-options): Compilation options.
  122. ### Meta parameters
  123. These meta parameters are special parameters (or properties) which can apply to
  124. any resource. The usefulness of doing so will depend on the particular meta
  125. parameter and resource combination.
  126. #### AutoEdge
  127. Boolean. Should we generate auto edges for this resource?
  128. #### AutoGroup
  129. Boolean. Should we attempt to automatically group this resource with others?
  130. #### Noop
  131. Boolean. Should the Apply portion of the CheckApply method of the resource
  132. make any changes? Noop is a concatenation of no-operation.
  133. #### Retry
  134. Integer. The number of times to retry running the resource on error. Use -1 for
  135. infinite. This currently applies for both the Watch operation (which can fail)
  136. and for the CheckApply operation. While they could have separate values, I've
  137. decided to use the same ones for both until there's a proper reason to want to
  138. do something differently for the Watch errors.
  139. #### Delay
  140. Integer. Number of milliseconds to wait between retries. The same value is
  141. shared between the Watch and CheckApply retries. This currently applies for both
  142. the Watch operation (which can fail) and for the CheckApply operation. While
  143. they could have separate values, I've decided to use the same ones for both
  144. until there's a proper reason to want to do something differently for the Watch
  145. errors.
  146. #### Poll
  147. Integer. Number of seconds to wait between `CheckApply` checks. If this is
  148. greater than zero, then the standard event based `Watch` mechanism for this
  149. resource is replaced with a simple polling mechanism. In general, this is not
  150. recommended, unless you have a very good reason for doing so.
  151. Please keep in mind that if you have a resource which changes every `I` seconds,
  152. and you poll it every `J` seconds, and you've asked for a converged timeout of
  153. `K` seconds, and `I <= J <= K`, then your graph will likely never converge.
  154. When polling, the system detects that a resource is not converged if its
  155. `CheckApply` method returns false. This allows a resource which changes every
  156. `I` seconds, and which is polled every `J` seconds, and with a converged timeout
  157. of `K` seconds to still converge when `J <= K`, as long as `I > J || I > K`,
  158. which is another way of saying that if the resource finally settles down to give
  159. the graph enough time, it can probably converge.
  160. #### Limit
  161. Float. Maximum rate of `CheckApply` runs started per second. Useful to limit
  162. an especially _eventful_ process from causing excessive checks to run. This
  163. defaults to `+Infinity` which adds no limiting. If you change this value, you
  164. will also need to change the `Burst` value to a non-zero value. Please see the
  165. [rate](https://godoc.org/golang.org/x/time/rate) package for more information.
  166. #### Burst
  167. Integer. Burst is the maximum number of runs which can happen without invoking
  168. the rate limiter as designated by the `Limit` value. If the `Limit` is not set
  169. to `+Infinity`, this must be a non-zero value. Please see the
  170. [rate](https://godoc.org/golang.org/x/time/rate) package for more information.
  171. #### Sema
  172. List of string ids. Sema is a P/V style counting semaphore which can be used to
  173. limit parallelism during the CheckApply phase of resource execution. Each
  174. resource can have `N` different semaphores which share a graph global namespace.
  175. Each semaphore has a maximum count associated with it. The default value of the
  176. size is 1 (one) if size is unspecified. Each string id is the unique id of the
  177. semaphore. If the id contains a trailing colon (:) followed by a positive
  178. integer, then that value is the max size for that semaphore. Valid semaphore
  179. id's include: `some_id`, `hello:42`, `not:smart:4` and `:13`. It is expected
  180. that the last bare example be only used by the engine to add a global semaphore.
  181. #### Rewatch
  182. Boolean. Rewatch specifies whether we re-run the Watch worker during a graph
  183. swap if it has errored. When doing a graph compare to swap the graphs, if this
  184. is true, and this particular worker has errored, then we'll remove it and add it
  185. back as a new vertex, thus causing it to run again. This is different from the
  186. `Retry` metaparam which applies during the normal execution. It is only when
  187. this is exhausted that we're in permanent worker failure, and only then can we
  188. rely on this metaparam.
  189. #### Realize
  190. Boolean. Realize ensures that the resource is guaranteed to converge at least
  191. once before a potential graph swap removes or changes it. This guarantee is
  192. useful for fast changing graphs, to ensure that the brief creation of a resource
  193. is seen. This guarantee does not prevent against the engine quitting normally,
  194. and it can't guarantee it if the resource is blocked because of a failed
  195. pre-requisite resource.
  196. *XXX: This is currently not implemented!*
  197. #### Reverse
  198. Boolean. Reverse is a property that some resources can implement that specifies
  199. that some "reverse" operation should happen when that resource "disappears". A
  200. disappearance happens when a resource is defined in one instance of the graph,
  201. and is gone in the subsequent one. This disappearance can happen if it was
  202. previously in an if statement that then becomes false.
  203. This is helpful for building robust programs with the engine. The engine adds a
  204. "reversed" resource to that subsequent graph to accomplish the desired "reverse"
  205. mechanics. The specifics of what this entails is a property of the particular
  206. resource that is being "reversed".
  207. It might be wise to combine the use of this meta parameter with the use of the
  208. `realize` meta parameter to ensure that your reversed resource actually runs at
  209. least once, if there's a chance that it might be gone for a while.
  210. ### Lang metadata file
  211. Any module *must* have a metadata file in its root. It must be named
  212. `metadata.yaml`, even if it's empty. You can specify zero or more values in yaml
  213. format which can change how your module behaves, and where the `mcl` language
  214. looks for code and other files. The most important top level keys are: `main`,
  215. `path`, `files`, and `license`.
  216. #### Main
  217. The `main` key points to the default entry point of your code. It must be a
  218. relative path if specified. If it's empty it defaults to `main.mcl`. It should
  219. generally not be changed. It is sometimes set to `main/main.mcl` if you'd like
  220. your modules code out of the root and into a child directory for cases where you
  221. don't plan on having a lot deeper imports relative to `main.mcl` and all those
  222. files would clutter things up.
  223. #### Path
  224. The `path` key specifies the modules import search directory to use for this
  225. module. You can specify this if you'd like to vendor something for your module.
  226. In general, if you use it, please use the convention: `path/`. If it's not
  227. specified, you will default to the parent modules directory.
  228. #### Files
  229. The `files` key specifies some additional files that will get included in your
  230. deploy. It defaults to `files/`.
  231. #### License
  232. The `license` key allows you to specify a license for the module. Please specify
  233. one so that everyone can enjoy your code! Use a "short license identifier", like
  234. `LGPLv3+`, or `MIT`. The former is a safe choice if you're not sure what to use.
  235. ### Graph definition file
  236. graph.yaml is the compiled graph definition file. The format is currently
  237. undocumented, but by looking through the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples/yaml/)
  238. you can probably figure out most of it, as it's fairly intuitive. It's not
  239. recommended that you use this, since it's preferable to write code in the
  240. [mcl language](language-guide.md) front-end.
  241. ### Command line
  242. The main interface to the `mgmt` tool is the command line. For the most recent
  243. documentation, please run `mgmt --help`.
  244. #### `--converged-timeout <seconds>`
  245. Exit if the machine has converged for approximately this many seconds.
  246. #### `--max-runtime <seconds>`
  247. Exit when the agent has run for approximately this many seconds. This is not
  248. generally recommended, but may be useful for users who know what they're doing.
  249. #### `--noop`
  250. Globally force all resources into no-op mode. This also disables the export to
  251. etcd functionality, but does not disable resource collection, however all
  252. resources that are collected will have their individual noop settings set.
  253. #### `--sema <size>`
  254. Globally add a counting semaphore of this size to each resource in the graph.
  255. The semaphore will get given an id of `:size`. In other words if you specify a
  256. size of 42, you can expect a semaphore if named: `:42`. It is expected that
  257. consumers of the semaphore metaparameter always include a prefix to avoid a
  258. collision with this globally defined semaphore. The size value must be greater
  259. than zero at this time. The traditional non-parallel execution found in config
  260. management tools such as `Puppet` can be obtained with `--sema 1`.
  261. #### `--allow-interactive`
  262. Allow interactive prompting for SSH passwords if there is no authentication
  263. method that works.
  264. #### `--ssh-priv-id-rsa`
  265. Specify the path for finding SSH keys. This defaults to `~/.ssh/id_rsa`. To
  266. never use this method of authentication, set this to the empty string.
  267. #### `--cconns`
  268. The maximum number of concurrent remote ssh connections to run. This defaults
  269. to `0`, which means unlimited.
  270. #### `--no-caching`
  271. Don't allow remote caching of the remote execution binary. This will require
  272. the binary to be copied over for every remote execution, but it limits the
  273. likelihood that there is leftover information from the configuration process.
  274. #### `--prefix <path>`
  275. Specify a path to a custom working directory prefix. This directory will get
  276. created if it does not exist. This usually defaults to `/var/lib/mgmt/`. This
  277. can't be combined with the `--tmp-prefix` option. It can be combined with the
  278. `--allow-tmp-prefix` option.
  279. #### `--tmp-prefix`
  280. If this option is specified, a temporary prefix will be used instead of the
  281. default prefix. This can't be combined with the `--prefix` option.
  282. #### `--allow-tmp-prefix`
  283. If this option is specified, we will attempt to fall back to a temporary prefix
  284. if the primary prefix couldn't be created. This is useful for avoiding failures
  285. in environments where the primary prefix may or may not be available, but you'd
  286. like to try. The canonical example is when running `mgmt` with remote execution
  287. there might be a cached copy of the binary in the primary prefix, but if there's
  288. no binary available continue working in a temporary directory to avoid failure.
  289. ### Compilation options
  290. You can control some compilation variables by using environment variables.
  291. #### Disable libvirt support
  292. If you wish to compile mgmt without libvirt, you can use the following command:
  293. ```
  294. GOTAGS=novirt make build
  295. ```
  296. #### Disable augeas support
  297. If you wish to compile mgmt without augeas support, you can use the following
  298. command:
  299. ```
  300. GOTAGS=noaugeas make build
  301. ```
  302. #### Disable docker support
  303. If you wish to compile mgmt without docker support, you can use the following
  304. command:
  305. ```
  306. GOTAGS=nodocker make build
  307. ```
  308. #### Combining compile-time flags
  309. You can combine multiple tags by using a space-separated list:
  310. ```
  311. GOTAGS="noaugeas novirt nodocker" make build
  312. ```
  313. ## Examples
  314. For example configurations, please consult the [examples/](https://github.com/purpleidea/mgmt/tree/master/examples)
  315. directory in the git source repository. It is available from:
  316. [https://github.com/purpleidea/mgmt/tree/master/examples](https://github.com/purpleidea/mgmt/tree/master/examples)
  317. ### Systemd:
  318. See [`misc/mgmt.service`](misc/mgmt.service) for a sample systemd unit file.
  319. This unit file is part of the RPM.
  320. To specify your custom options for `mgmt` on a systemd distro:
  321. ```bash
  322. sudo mkdir -p /etc/systemd/system/mgmt.service.d/
  323. cat > /etc/systemd/system/mgmt.service.d/env.conf <<EOF
  324. # Environment variables:
  325. MGMT_SEEDS=http://127.0.0.1:2379
  326. MGMT_CONVERGED_TIMEOUT=-1
  327. MGMT_MAX_RUNTIME=0
  328. # Other CLI options if necessary.
  329. #OPTS="--max-runtime=0"
  330. EOF
  331. sudo systemctl daemon-reload
  332. ```
  333. ## Development
  334. This is a project that I started in my free time in 2013. Development is driven
  335. by all of our collective patches! Dive right in, and start hacking!
  336. Please contact me if you'd like to invite me to speak about this at your event.
  337. You can follow along [on my technical blog](https://purpleidea.com/blog/).
  338. To report any bugs, please file a ticket at: [https://github.com/purpleidea/mgmt/issues](https://github.com/purpleidea/mgmt/issues).
  339. ## Authors
  340. Copyright (C) 2013-2021+ James Shubin and the project contributors
  341. Please see the
  342. [AUTHORS](https://github.com/purpleidea/mgmt/tree/master/AUTHORS) file
  343. for more information.
  344. * [github](https://github.com/purpleidea/)
  345. * [&#64;purpleidea](https://twitter.com/#!/purpleidea)
  346. * [https://purpleidea.com/](https://purpleidea.com/)