Back

FreeBSD::Portindex

Introduction

portindex, portdepends, cache-init, cache-update, find-updated and make0readmes are a set of perl scripts built around the common core of the FreeBSD::Portindex perl modules. Their use is to generate and maintain the ports INDEX or INDEX-N files speedily and efficiently (where N is one of 7, 8, 9 or 10 depending on FreeBSD major version number). Ultimately they work in a very similar way to the standard make index command, except that the FreeBSD::Portindex tools keep a cache of the make describe output from each port and timestamps for Makefiles and pkg-descr files, and can update that cached data incrementally as the ports tree itself is updated.

Incremental updating is much faster than building the INDEX entirely from scratch every time. Once the cache has been initialised, all of the changes made over the course of a day or a week can be incorporated in a few minutes, and an up to date INDEX file produced in a few seconds. Being able to quickly generate a fresh INDEX file from the cache is also exceedingly handy.

Why would you build your own version of the INDEX rather than just using make fetchindex to download a recently generated default version or taking the INDEX file produced by portsnap? Firstly: because your locally built INDEX will reflect the configuration choices you've made either through the OPTIONS dialogues that many ports use, or by editing /etc/make.conf. Secondly: because portindex gives you an easy way to incorporate information about local ports into your INDEX.

The cache-update process will now detect modifications to any files read during the generation of the INDEX and update the cached data for the affected ports. No other input is required, although you can still update the cache the old way; by parsing the output of running svn or the output of portsnap update. All of these methods serve to work out which ports have been modified or deleted recently — this includes ports that have been affected indirectly by a change to any of the ports Makefiles involved in building them, and so need their cached description updated.

find-updated has been radically rewritten. Rather than running find over the ports tree, now it simply compares timestamps in exactly the same way as cache-update. It's really just a read-only preview of what cache-update would do.

The portdepends script prints out the dependency tree for a port: recent discussion on the freebsd-ports mailing list suggested that there weren't many good tools available to do this. FreeBSD::Portindex already had 90% of the code required, so adding this was a no-brainer.

Vizualizing dependency graphs is not easily done from the original textual output of portdepends, so with version 2.5, portdepends has acquired the ability to print out the dependency graph in GraphViz dot generic input format, allowing pictorial representations of dependency trees to be easily generated. Here is the a fairly simple example: the dependency graph for portindex itself.

make-readmes does exactly what it says on the tin. It makes a tree of README.html files — one for each port or category and one at the top of the tree. The result should be pretty much equivalent to running make readme in $PORTSDIR (the generated html in particular is about as bad as the original.) Advantages of the make-readmes script are speed and that it can create the tree of README.html files in an arbitrary directory, not just based at $PORTSDIR.

Installation and Operation — Quick Start

Please see the POD documentation for each of the commands for details. However, routine usage will typically be something like this:

0) Dependencies

Prerequisites: you will need to have installed a recent version of Perl, preferably perl-5.16.2 (lang/perl5.16) or later. Additionally you will need p5-BerkeleyDB-0.51 (databases/p5-BerkeleyDB) and one of the Berkeley Database Library ports such as db5-5.3.28 (databases/db5). Other Perl or BDB versions will work, but have not been thoroughly tested.

Optionally you can also install graphviz-2.28.0_1 (graphics/graphviz), but this is only required to render the portdepends output into an image format; not for portdepends itself to run.

i) Installation

Of course, installing via the ports-mgmt/p5-FreeBSD-Portindex port is the recommended method.

However, for completeness' sake, here are the instructions for how to do a manual installation. Download and unpack the portindex tar-ball, and install:

        % tar -xvf FreeBSD-Portindex-N.NN.tar.xz
        % cd FreeBSD-Portindex-N.NN
        % perl Makefile.PL
        % make
        % make test
        # make install
        % rehash
    

ii) Initialise the Cache

This is a slow step (slower than make index) but it only needs to be done occasionally.

        # mkdir /var/db/portindex
        # chown user:group /var/db/portindex
        % cache-init
    

That creates the portindex cache file (about 55MiB currently) containing all of the make describe output, and timestamp data for all the various Makefiles and pkg-descr files required to generate the INDEX-N entry for the port. It also contains a timestamp file showing the last time that the cache was updated. Nb. while you can run any of the FreeBSD::Portindex programs as root safely enough, on general principles I recommend using a non-privileged user account instead.

Then, for day to day use:

iii) Update the Cache

Before generating a new INDEX-N file, update the FreeBSD::Portindex cache with the details of any recently updated ports. The recommended way to do this is by comparing the timestamps recorded in the cache with the current state of what is on disk.

      % cache-update
    

The old methods of parsing the output of csup or portsnap update, and the various other alternatives still work. See the POD pages for cache-update and find-updated for more information.

Repeat as necessary if you update the ports tree several times between generating new INDEX-N files.

iv) Generate the INDEX-N file

       # portindex -o /usr/ports/INDEX-N
    

Steps (iii) and (iv) together should take a minute or so — more if one of the commonly included Makefiles like /usr/ports/Mk/bsd.gnome.mk has been modified. Even so, it will be very much faster than building the INDEX-N file from scratch each time you update your ports.

Downloads

The latest version is FreeBSD::Portindex-3.3 released on 26th December 2013.

Available versions

Version Size Date Comments
3.3 56192 2013/12/26 Drop all support for old style OPTIONS, account for variant locations where options may be stored.
3.2 56036 2013/03/31 Really fix behaviour where /usr/ports is a sym-link.
3.1 55876 2013/03/30 Drop cvsup-output and cvsup-checkouts input formats, add svn-up. Fix behaviour where /usr/ports is a sym-link.
3.0 57096 2012/12/02 Detect ports that use options correctly with OPTIONSng. Switch to pragma use parent for inheritance
2.9 56728 2012/06/01 Adjust Endemic and Ubiquitous Makefile settings to account for OPTIONSng
2.8 56240 2012/02/25 Radical redisgn of how the cache operates
2.7 52612 2011/11/07 Bugfixes, documentation improvements, fix dependency tracking (again!)
2.6 51828 2011/10/26 Fix bug that could crash portindex.
2.5 51700 2011/10/07 GraphViz mode for portdepends.
2.4 50724 2011/08/28 1) portdepends short output mode. 2) Bugfix handling LIB_DEPENDS and RUN_DEPENDS.
2.3 50095 2011/03/19 New portdepends script, update EndemicMakefiles setting.
2.2 46188 2009/08/09 Object inheritance reorganisation, add alternate SHLIBS index format output for portindex.
2.1 32939 2008/04/07 Extend the plain mode to read portsnap output.
2.0 32206 2007/08/05 OPTIONS processing, strict mode, reworked internal data structures.
1.9 29297 2007/02/04 DEPENDS variable and supporting code removed from bsd.ports.mk
1.8 29342 2006/09/17 Fix crash when handling broken dependency information.
1.7 29110 2006/07/23 Improved handling of Master/Slave relations between ports. Don't quit on errors.
1.6 27889 2006/06/25 Handle Makefile.local additions to ports. Improve handing of port category changes.
1.5 24514 2006/01/29 Fix handling of empty port categories when running cache-init.
1.4 24390 2005/02/19 Fix recursion loop, enable processing cvsup-cvs mode output.
1.3 24030 2005/01/21 Fix programming mistake that prevented cache-update's cvsup-checkouts mode from working.
1.2 23975 2005/01/16 Respect $PORTSDIR. Make some Makefile lists configurable.
1.1 21958 2004/11/05 Add processing based on included makefiles.
1.0 19366 2004/11/01 First public release
0.3 18602 2004/10/30 BETA - Reorganise class hierarchy, add find-updated
0.2 15899 2004/10/23 BETA - Bug fixes
0.1 15850 2004/10/22 BETA - First publicly available version

Also available: SHA256 Checksums for all available release versions

Bugs

Please report any bugs to me, Matthew Seaman with as much detail as possible so I can try and reproduce them. Please try and include:

FreeBSD version — uname -a output.

Perl version — perl -V output.

Transcript of session showing how the problem occurs.

As well as anything else you think is relevant.