Page MenuHomeSoftware Heritage

Manage the parent directories of the kafka logdirs
ClosedPublic

Authored by vsellier on Tue, Nov 17, 6:20 PM.

Details

Summary

In staging and vagrant vms, the logdir is not stored on a mountpoint
This commit will allow avoid manual actions when it's not needed

Related to T2790

Test Plan
  • octocatalog-diff for journal0.internal.staging.swh.network
diff origin/production/journal0.internal.staging.swh.network current/journal0.internal.staging.swh.network
*******************************************
+ Exec[create /srv/kafka/logdir] =>
   parameters =>
     "command": "mkdir -p /srv/kafka/logdir",
     "creates": "/srv/kafka/logdir",
     "path": [
       "/bin",
       "/usr/bin",
       "/sbin",
       "/usr/sbin"
     ]
*******************************************
*** End octocatalog-diff on journal0.internal.staging.swh.network

*octocatalog-diff for kafka1.internal.softwareheritage.org

diff origin/production/kafka1.internal.softwareheritage.org current/kafka1.internal.softwareheritage.org
*******************************************
+ Exec[create /srv/kafka/logdir] =>
   parameters =>
     "command": "mkdir -p /srv/kafka/logdir",
     "creates": "/srv/kafka/logdir",
     "path": [
       "/bin",
       "/usr/bin",
       "/sbin",
       "/usr/sbin"
     ]
*******************************************
*** End octocatalog-diff on kafka1.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17405
Build 26891: arc lint + arc unit

Event Timeline

vsellier created this revision.Tue, Nov 17, 6:20 PM
ardumont accepted this revision.Tue, Nov 17, 7:20 PM
This revision is now accepted and ready to land.Tue, Nov 17, 7:20 PM

I will not land this now, it seems there is another issue with the startup of kafka when the logdir is already existing but empty (i.e. created by puppet). I need to dig further

[2020-11-19 10:22:02,563] ERROR Failed to create or validate data directory /srv/kafka/logdir (kafka.server.LogDirFailureChannel)
java.io.IOException: Failed to load /srv/kafka/logdir during broker startup
        at kafka.log.LogManager.$anonfun$createAndValidateLogDirs$1(LogManager.scala:150)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:553)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:551)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:920)
        at kafka.log.LogManager.createAndValidateLogDirs(LogManager.scala:147)
        at kafka.log.LogManager.<init>(LogManager.scala:81)
        at kafka.log.LogManager$.apply(LogManager.scala:1111)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:273)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:82)
        at kafka.Kafka.main(Kafka.scala)
[2020-11-19 10:22:02,566] ERROR Shutdown broker because none of the specified log dirs from /srv/kafka/logdir can be created or validated (kafka.log.LogManager)
vsellier updated this revision to Diff 16200.Mon, Nov 23, 3:46 PM

Avoid the declaration of the parent directories by using mkdir -p

vsellier requested review of this revision.Mon, Nov 23, 3:46 PM

That's often the behavior i'm looking for when i'm using the "file" instruction on a deep arborescence...

ardumont accepted this revision.Mon, Nov 23, 4:55 PM
This revision is now accepted and ready to land.Mon, Nov 23, 4:55 PM
vsellier edited the test plan for this revision. (Show Details)Mon, Nov 23, 5:27 PM