Page MenuHomeSoftware Heritage
Differential D8777

Puppet: Lister implements incremental mode
ClosedPublic
Actions

Authored by franckbret on Oct 26 2022, 10:14 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDLSe1f3f87c73f4: Puppet: Lister implements incremental mode
Summary

Use with_release_since api argument to retrieve modules that have been
updated since the last date the lister has been executed.

Related T4519

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

franckbret created this revision.Oct 26 2022, 10:14 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 26 2022, 10:14 AM
swh-public-ci added a comment.Oct 26 2022, 10:20 AM

Build is green

Patch application report for D8777 (id=31637)

Rebasing onto 8355fee25f...

Current branch diff-target is up to date.
Changes applied before test
commit 8a026b816cc977d4bf1718b3057a9253a4eb3e25
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 26 10:09:03 2022 +0200

    Puppet: Lister implements incremental mode
    
    Use with_release_since api argument to retrieve modules that have been
    updated since the last date the lister has been executed.
    
    Related T4519

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/834/ for more details.

Harbormaster completed remote builds in B32590: Diff 31637.Oct 26 2022, 10:20 AM
franckbret requested review of this revision.Oct 26 2022, 10:20 AM
vlorentz added a subscriber: vlorentz.Nov 2 2022, 12:18 PM
vlorentz added inline comments.
swh/lister/puppet/lister.py
83–85

If I understand this correctly:

Constrain results to modules that have had at least one release since the given ISO 8601 date

then incremental loads would miss all packages updated later in the (local) day of the previous visit.

Did you check with_release_since is inclusive?

vlorentz added inline comments.Nov 2 2022, 12:21 PM
swh/lister/puppet/lister.py
83–85

also, what is the timezone used by with_release_since?
The timestamps I see in the file below are American, so it might not be UTC.

franckbret added inline comments.Nov 3 2022, 11:56 AM
swh/lister/puppet/lister.py
83–85

Yes it is inclusive and It returns the same results given a date or datetime,

Query with a datetime

############################################################               
Two days ago:  2022-11-01T11:12:14.744560                                                                 
############################################################
2022-11-01 03:13:02 -0700                           
2022-11-01 04:57:26 -0700                           
2022-11-01 05:57:58 -0700                           
2022-11-01 10:06:31 -0700                                                                                
2022-11-01 10:28:20 -0700                                                                                
2022-11-01 14:53:51 -0700                           
2022-11-02 02:38:13 -0700                           
2022-11-02 03:32:59 -0700                           
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700                           
2022-11-02 09:25:05 -0700                                                                                
2022-11-02 14:56:51 -0700                                                                                
2022-11-02 17:57:44 -0700                                                                                
2022-11-02 19:20:00 -0700                                                                                
2022-11-02 20:13:51 -0700                                                                                
2022-11-03 02:24:39 -0700                           
2022-11-03 02:31:19 -0700                           
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700                                                                                
############################################################
One day ago:  2022-11-02T11:12:14.744560                                                                       
############################################################       
2022-11-02 02:38:13 -0700                           
2022-11-02 03:32:59 -0700                           
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700                           
2022-11-02 09:25:05 -0700                           
2022-11-02 14:56:51 -0700                           
2022-11-02 17:57:44 -0700                                                                                
2022-11-02 19:20:00 -0700                                                                                
2022-11-02 20:13:51 -0700                                                                                
2022-11-03 02:24:39 -0700                           
2022-11-03 02:31:19 -0700                           
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700 

Query with a date 

############################################################
Two days ago:  2022-11-01                            
############################################################
2022-11-01 03:13:02 -0700                           
2022-11-01 04:57:26 -0700                           
2022-11-01 05:57:58 -0700                           
2022-11-01 10:06:31 -0700                           
2022-11-01 10:28:20 -0700                           
2022-11-01 14:53:51 -0700                           
2022-11-02 02:38:13 -0700                           
2022-11-02 03:32:59 -0700                           
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700                           
2022-11-02 09:25:05 -0700                           
2022-11-02 14:56:51 -0700                           
2022-11-02 17:57:44 -0700                           
2022-11-02 19:20:00 -0700                           
2022-11-02 20:13:51 -0700                           
2022-11-03 02:24:39 -0700                           
2022-11-03 02:31:19 -0700                           
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700                           
############################################################
One day ago:  2022-11-02                                  
############################################################
2022-11-02 02:38:13 -0700                           
2022-11-02 03:32:59 -0700                           
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700                           
2022-11-02 09:25:05 -0700                           
2022-11-02 14:56:51 -0700                           
2022-11-02 17:57:44 -0700                           
2022-11-02 19:20:00 -0700                           
2022-11-02 20:13:51 -0700                           
2022-11-03 02:24:39 -0700                           
2022-11-03 02:31:19 -0700                           
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700
83–85

Looks like the query is UTC, it returns the same results when specifying an offset or utc

With -7 offset

############################################################                                             
Two days ago:  2022-11-01T03:37:55.576098-07:00      
############################################################                                             
2022-11-01 03:13:02 -0700
2022-11-01 04:57:26 -0700                           
2022-11-01 05:57:58 -0700                           
2022-11-01 10:06:31 -0700                                                                                
2022-11-01 10:28:20 -0700 
2022-11-01 14:53:51 -0700                                                                                
2022-11-02 02:38:13 -0700                           
2022-11-02 03:32:59 -0700                           
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700                                                                                
2022-11-02 09:25:05 -0700                                                                                
2022-11-02 14:56:51 -0700                           
2022-11-02 17:57:44 -0700                                                                                
2022-11-02 19:20:00 -0700                                                                                
2022-11-02 20:13:51 -0700                           
2022-11-03 02:24:39 -0700                                                                                
2022-11-03 02:31:19 -0700                                                                                
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700 
############################################################                                             
One day ago:  2022-11-02T03:37:55.576098-07:00                                                                 
############################################################                                             
2022-11-02 02:38:13 -0700                                                                                
2022-11-02 03:32:59 -0700                                                                                
2022-11-02 04:53:04 -0700                           
2022-11-02 09:23:28 -0700
2022-11-02 09:25:05 -0700
2022-11-02 14:56:51 -0700                           
2022-11-02 17:57:44 -0700                           
2022-11-02 19:20:00 -0700                                                                                
2022-11-02 20:13:51 -0700 
2022-11-03 02:24:39 -0700                                                                                
2022-11-03 02:31:19 -0700                           
2022-11-03 02:47:49 -0700                           
2022-11-03 02:51:34 -0700 

With utc 

############################################################
Two days ago:  2022-11-01T10:50:35.671332+00:00
############################################################
2022-11-01 03:13:02 -0700
2022-11-01 04:57:26 -0700
2022-11-01 05:57:58 -0700
2022-11-01 10:06:31 -0700
2022-11-01 10:28:20 -0700
2022-11-01 14:53:51 -0700
2022-11-02 02:38:13 -0700
2022-11-02 03:32:59 -0700
2022-11-02 04:53:04 -0700
2022-11-02 09:23:28 -0700
2022-11-02 09:25:05 -0700
2022-11-02 14:56:51 -0700
2022-11-02 17:57:44 -0700
2022-11-02 19:20:00 -0700
2022-11-02 20:13:51 -0700
2022-11-03 02:24:39 -0700
2022-11-03 02:31:19 -0700
2022-11-03 02:47:49 -0700
2022-11-03 02:51:34 -0700
############################################################
One day ago:  2022-11-02T10:50:35.671332+00:00
############################################################
2022-11-02 02:38:13 -0700
2022-11-02 03:32:59 -0700
2022-11-02 04:53:04 -0700
2022-11-02 09:23:28 -0700
2022-11-02 09:25:05 -0700
2022-11-02 14:56:51 -0700
2022-11-02 17:57:44 -0700
2022-11-02 19:20:00 -0700
2022-11-02 20:13:51 -0700
2022-11-03 02:24:39 -0700
2022-11-03 02:31:19 -0700
2022-11-03 02:47:49 -0700
2022-11-03 02:51:34 -0700
vlorentz added inline comments.Nov 3 2022, 1:37 PM
swh/lister/puppet/lister.py
83–85

So, assume we run the lister at 2022-11-02 01:00:00 UTC, aka 2022-11-01 18:00:00 -0700. The next run of the lister will query with with_release_since=2022-11-02, which returns all results after 2022-11-02 00:00:00 -0700, according to your last two examples.

So the lister will miss origins updated between 2022-11-01 18:00:00 -0700 and 2022-11-01 24:00:00 -0700

franckbret updated this revision to Diff 31742.Nov 4 2022, 10:15 AM

Ensure we query the api with the same timezone Us/Pacific date as the http api use for querying and expressing results

franckbret marked 3 inline comments as done.Nov 4 2022, 10:17 AM
franckbret added inline comments.
swh/lister/puppet/lister.py
83–85

Right, I improved the code to ensure we query the http api with an aware date

swh-public-ci added a comment.Nov 4 2022, 10:20 AM

Build is green

Patch application report for D8777 (id=31742)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: Puppet: Lister implements incremental mode
Changes applied before test
commit df1790caa297bc409f147a2a58ac577874e99f63
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 26 10:09:03 2022 +0200

    Puppet: Lister implements incremental mode
    
    Use with_release_since api argument to retrieve modules that have been
    updated since the last date the lister has been executed.
    
    Related T4519

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/840/ for more details.

Harbormaster completed remote builds in B32697: Diff 31742.Nov 4 2022, 10:21 AM
vlorentz added a comment.EditedNov 4 2022, 11:00 AM

State storage is not guaranteed to return the same timezone that was written (postgresql returns the local timezone of the server iirc). You should convert it just before querying the API.

Also, hardcoding -7 is not a solution. First, because the API will probably switch from UTC-7 to UTC-8 next Sunday (as the US enter Winter Time). More generally, the API does not guarantee it won't change timezones in the future (especially as the API designers didn't seem to consider timezones at all).

So just to be safe, you should remove 15h, which is the lowest timezone recorded in the tzdb.

franckbret updated this revision to Diff 31746.Nov 4 2022, 11:56 AM
franckbret marked an inline comment as done.

Use an offset of -15h when querying the api which is the lower timezone recorded in the tzdb

swh-public-ci added a comment.Nov 4 2022, 12:02 PM

Build is green

Patch application report for D8777 (id=31746)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: Puppet: Lister implements incremental mode
Changes applied before test
commit 5dd180652e4cda7c68b21b28bc1653dc6be0fa91
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 26 10:09:03 2022 +0200

    Puppet: Lister implements incremental mode
    
    Use with_release_since api argument to retrieve modules that have been
    updated since the last date the lister has been executed.
    
    Related T4519

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/841/ for more details.

Harbormaster completed remote builds in B32701: Diff 31746.Nov 4 2022, 12:02 PM
vlorentz accepted this revision.Nov 8 2022, 10:55 AM
This revision is now accepted and ready to land.Nov 8 2022, 10:55 AM
franckbret updated this revision to Diff 31805.Nov 8 2022, 2:33 PM

Rebase

This revision was landed with ongoing or failed builds.Nov 8 2022, 2:34 PM
Closed by commit rDLSe1f3f87c73f4: Puppet: Lister implements incremental mode (authored by franckbret). · Explain Why
This revision was automatically updated to reflect the committed changes.
franckbret added a commit: rDLSe1f3f87c73f4: Puppet: Lister implements incremental mode.
swh-public-ci added a comment.Nov 8 2022, 2:38 PM

Build is green

Patch application report for D8777 (id=31805)

Rebasing onto e8699422d7...

Current branch diff-target is up to date.
Changes applied before test
commit e1f3f87c73f4ea25e06bbe3591bd54a9675b1785
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 26 10:09:03 2022 +0200

    Puppet: Lister implements incremental mode
    
    Use with_release_since api argument to retrieve modules that have been
    updated since the last date the lister has been executed.
    
    Related T4519

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/854/ for more details.

Harbormaster completed remote builds in B32746: Diff 31805.Nov 8 2022, 2:38 PM

Revision Contents
Changeset List

PathSize
swh/
lister/
puppet/
4 lines
54 lines
tests/
data/
https_forgeapi.puppet.com/
286 lines
44 lines

Diff 31806

View Options

swh/lister/puppet/__init__.py

Loading...
View Options

swh/lister/puppet/lister.py

Loading...
View Options

swh/lister/puppet/tests/data/https_forgeapi.puppet.com/v3_modules,limit=100,with_release_since=2022-09-26

Loading...
View Options

swh/lister/puppet/tests/test_lister.py

Loading...
About · Developer information · Legal