Page MenuHomeSoftware Heritage

plugin to monitor the amount of visits per origin
Open, NormalPublic

Description

Similar to T361, it would be nice to monitor the amount of visits that each origin gets.
The query (this time on the regular SWH DB) is the following one (see P51):

softwareheritage=> with origin_visits as (
     select origin, count(visit) as visits
     from origin_visit
     group by origin
     )
select count(origin), visits
from origin_visits
group by visits
order by visits desc;
  count  | visits 
---------+--------
       7 |      5
    2996 |      4
 5985138 |      3
 4853579 |      2
  956040 |      1

Event Timeline

zack created this task.Apr 1 2016, 5:03 PM
olasd changed the visibility from "All Users" to "Public (No Login Required)".May 13 2016, 5:09 PM
fero added a subscriber: fero.Jan 29 2018, 2:11 AM

Hi all guys and many compliments for your work!

I have just listened to the FOSDEM 2017 KeyNote and I would like to contribute to an easy hack. Is this still helpful?

I have seen the T361 and it would have been very helpful to me to see the code of the munin plugin has been developed to solve that task.
But surprisingly... I haven't find the munin plugin in the softwareheritage repositories!!! What the hell is going on?!? ;-)

Furthermore the url provided for the output of the plugin referred by the @zack comment https://forge.softwareheritage.org/T361#5160 is not reachable from .. I think... outside your network?!? I mean the host munin.internal.softwareheritage.org which is unresolvable from public DNS.

In T362#17396, @fero wrote:

Hi all guys and many compliments for your work!

Hi Fero! … and sure, contributions are still and will always be welcome :)

I have seen the T361 and it would have been very helpful to me to see the code of the munin plugin has been developed to solve that task.
But surprisingly... I haven't find the munin plugin in the softwareheritage repositories!!! What the hell is going on?!? ;-)

The entry point for our puppet recipes is https://forge.softwareheritage.org/diffusion/SENV/ , see the README there.
And, sure enough, there is there a pointer to the munin module we use, which is ssm-munin.

Furthermore the url provided for the output of the plugin referred by the @zack comment https://forge.softwareheritage.org/T361#5160 is not reachable from .. I think... outside your network?!? I mean the host munin.internal.softwareheritage.org which is unresolvable from public DNS.

That's true, IIRC we discussed that it'd be totally fine to open up munin access to the world, but that just hasn't happened that. Let me punt to @olasd and @ftigeot here.

fero added a comment.Feb 1 2018, 2:23 AM

Hi zack!

I am a really newbie of puppet, the first goal for me now is finding already written plugin to inherit code and conventions and write the munin plugin to (hopefully) close this issue.

So, I have gone to the https://forge.softwareheritage.org/diffusion/SENV/ repo you pointed me and I have found the mention to ssm-munin in the .mrconfig line 196:
https://forge.softwareheritage.org/diffusion/SENV/browse/master/.mrconfig;118efccbfd4b7f5c94443fac554fe17fb877c0fa$196

Then I have uploaded my pubkey and cloned the repo

After that I expected to find some results in doing rgrep current_interval * (like in the query at T361) or to find some files with code for the munin plugin of T361 ... but ... didn't find :(

Where is the fault in my workflow in finding a file with the query of T361?

ardumont added a subscriber: ardumont.EditedJun 13 2018, 3:10 PM

Hello,

...
Where is the fault in my workflow in finding a file with the query of T361?

When i did one [1], i had to install manually the internal graph in munin instance.
I pushed them in our snippets repository, here is one for example.

So, i'd say none on your part really.

I am a really newbie of puppet, the first goal for me now is finding already written plugin to inherit code and conventions and write the munin plugin to (hopefully) close this issue.

There are some reasons [2] that makes us want to move away from munin towards the prometheus/grafana couple.
You would not be able to graph yet (grafana part is not puppetised up to the graph point, only install) but at least, but you could extract the data.
And this would be greatly helpful already ;)

Now, there would be some code sample [3] that could help in that direction.

Hope this helps.

[1] http://munin.internal.softwareheritage.org/softwareheritage.org/prado.softwareheritage.org/softwareheritage_scheduler_oneshot.html

[2] T988

[3] https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/data/hostname/somerset.internal.softwareheritage.org.yaml$8-26

Cheers,

ardumont renamed this task from munin plugin to monitor the amount of visits per origin to plugin to monitor the amount of visits per origin.Jun 13 2018, 3:10 PM