cpan.loader - archive Perl modules from CPAN
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	joenio
	Dec 1 2020, 11:33 AM

Description

Hi SWH folks,

I would like to know if is there any plan or work-in-progress to preserve Perl modules from CPAN.org? CPAN[1] has a incredible archive containing 196,752 Perl modules and 42,031 distributions, also CPAN has a tradition to archive old versions of all modules through BackPan[2], which increase still more the total number of modules, but I'm not sure how much.

I'm interested in collaborate in this but I have no experience on SWH dev stack, also have no idea how much effort is to add a new loader on SWH, then this task here is a first step in it to know if there is already any kind of cpan.loader or if makes sense to have it on SWH.

[1] - https://www.cpan.org
[2] - http://backpan.perl.org

Thanks,
Joenio Marques da Costa
CorTexT platform / RISIS2.eu project

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
	Closed		D8557 Cpan: Cpan loader loads Perl modules from cpan.org
		D8801	rDLDBASEbf2cb039d5d7 cpan: Add extid manifest to CpanPackageInfo
		D8652	rDLDBASE85963318aab6 cpan: Collect extrinsic metadata for each module release
		D8651	rDLDBASE7b929606a78f cpan: Do not parse intrinsic metadata for getting module author
		D8616	rDLDBASEa13e3e6f35bc cpan: Align loader implementation with latest lister improvements
rDSTO Storage manager
		D8689	rDSTO17d9ad235c3e docs: Add info about CPAN extrinsic metadata format
rDLS Listers
	Abandoned		D8649 cpan: Use a fake origin URL instead of an HTTP one
		D8648	rDLS05cd1de1cde7 cpan: Fix module version extraction for some edge cases
		D8615	rDLSf57b8f3a2c49 cpan: Improve listing process by querying the metacpan release endpoint
rDCORE Foundations and core functionalities
		D8661	rDCORE6a5ad7618587 tarball: fallback using tar command when shutil.unpack_archive failed

Related Objects
Search...

		Status	Assigned	Task
				Unknown Object (Maniphest Task)
		Migrated	gitlab-migration	T2833 cpan.loader - archive Perl modules from CPAN

Event Timeline

joenio triaged this task as Wishlist priority.Dec 1 2020, 11:33 AM

joenio created this task.

joenio created this object in space S1 Public.

ardumont added a project: Archive coverage.Dec 1 2020, 11:41 AM

Hi SWH devs,

Any answer here?

I would like to know if a cpan.loader is something desired, if so I'm interested contributing working on it.

Thanks,
Joenio Marques da Costa
CorTexT platform / RISIS2.eu project

Hey, yes, we want to have one, but nobody is working it at the moment, and we rather have someone knowledgeable with that ecosystem to work on it. So, if you're interested, you're more than welcome to help there! (And thank you in advance.)

zack renamed this task from [feature request] cpan.loader - preserver Perl modules from CPAN to cpan.loader - preserver Perl modules from CPAN.Mar 30 2021, 8:22 AM

zack raised the priority of this task from Wishlist to Normal.

Thanks @zack for the info, I'll start learning the SWH dev stack following the instructions I found in the wiki[1].

[1] https://docs.softwareheritage.org/devel/

awesome, thanks @joenio ! you can also drop by our other devel communication channel if you want to discuss this in other ways: https://www.softwareheritage.org/community/developers/

Hi @joenio

I've just joined SWH as an ambassador and wondered how you are getting on with the cpan.loader? Or maybe metacpan.loader now?

Thanks,
Gavin.

zack renamed this task from cpan.loader - preserver Perl modules from CPAN to cpan.loader - archive Perl modules from CPAN.Apr 22 2022, 11:26 AM

CPANTS These indicators might be helpful in categorizing incoming data. So that the mining process is significantly easier

It's what metacpan-api uses.

Would we care for instant update mirroring for package downloads and then we can use a sort of post download extractor.

In T2833#83806, @ghenry wrote:

Hi @joenio

I've just joined SWH as an ambassador and wondered how you are getting on with the cpan.loader? Or maybe metacpan.loader now?

Thanks,
Gavin.

Hi @ghenry

I haven't started to develop nothing for this loader.

About the name cpan.loader or metacpan.loader, my bet is that it should continue as cpan.loader cause metacpan is just a search engine, and a new UI on top of CPAN but the repository is still named as CPAN.

Sounds good.

bchauvet added a parent task: Unknown Object (Maniphest Task).Sep 2 2022, 11:00 AM

anlambert added a project: CPAN lister.Sep 9 2022, 3:52 PM

franckbret mentioned this in D8542: Cpan: List Perl module origins from cpan.org.Sep 27 2022, 9:00 AM

franckbret mentioned this in rDLSa4aec3894e3c: Cpan: List Perl module origins from cpan.org.Sep 27 2022, 2:32 PM

franckbret mentioned this in D8557: Cpan: Cpan loader loads Perl modules from cpan.org.Sep 28 2022, 9:41 AM

franckbret mentioned this in rDLDBASE2db1a754e368: Cpan: Cpan loader loads Perl modules from cpan.org.Sep 29 2022, 3:17 PM

bchauvet added a revision: D8557: Cpan: Cpan loader loads Perl modules from cpan.org.Sep 30 2022, 9:13 AM

anlambert added a revision: D8615: cpan: Improve listing process by querying the metacpan release endpoint.Oct 4 2022, 5:17 PM

anlambert added a revision: D8616: cpan: Align loader implementation with latest lister improvements.Oct 4 2022, 5:24 PM

anlambert added a revision: D8648: cpan: Fix module version extraction for some edge cases.Oct 10 2022, 4:26 PM

anlambert added a revision: D8649: cpan: Use a fake origin URL instead of an HTTP one.Oct 10 2022, 4:32 PM