• corsicanguppy@lemmy.ca
    link
    fedilink
    English
    arrow-up
    3
    ·
    2 days ago

    rebuilding internet facing servers

    Because of one vulnerability?

    Jebus. Response at one site was a 10-minute thing and then letting cron do the work.

    Where the hell is your D-R? I worry you’ve made some very non-optimal decisions in the past and are taking on way too much work now.

    • Matt The Horwood@lemmy.horwood.cloud
      link
      fedilink
      English
      arrow-up
      5
      ·
      2 days ago

      It took longer than planned as we are in the process of getting everything sorted for quick rebuilds, most of the time was waiting for GitHub to not poo it’s self.

      If we had everything in ansible it would be 5 minutes

      • corsicanguppy@lemmy.ca
        link
        fedilink
        English
        arrow-up
        1
        ·
        18 hours ago

        If we had everything in ansible it would be 5 minutes

        Ansible at the day-job takes about 45 min to do a patch. As someone who’s used a fair number of alternatives, maybe consider anything else on the planet?

        But, seriously. The response to this particular sploit wasn’t really “config management”-y. It was a simple process that cron generally handled anyway, and network segregation and firewalling allowed us to prioritize easily. My worry is that there are some core architectural decisions you may need to revisit in the post-mortem on this, because there may be some fundamental changes required to get sensible wins in the future.

        (If you’re on containers, I weep for your validation challenges; also, get bazel)

        • Matt The Horwood@lemmy.horwood.cloud
          link
          fedilink
          English
          arrow-up
          3
          ·
          17 hours ago

          We find ansible to be the best way to manage our fleet, patching and keeping config in sync.

          That’s not to say it can’t be slow sometimes, but it works for us.