Tracking NetBSD-current

Frequently asked questions

Why track NetBSD-current?
Installing a current snapshot
Downloading current source
Building a release from source
Updating an existing system
Things you need to remember
What are the various Makefile targets?
Using anoncvs
Using anoncvs over ssh
Tracking NetBSD-current with anoncvs
Tracking NetBSD-current with SUP into CVS
Supping sources
Importing and merging sources.
Building current.
Tagging a successful build
Getting the whole repository
What if I get an error?
Updating the configuration and startup files with etcupdate

Specific problems

Console dead after updating to wscons
Why does build.sh always rebuild nbmake first?

Frequently asked questions

Why track NetBSD-current? (top)

The developers of NetBSD have made the current development sources available to the public for several reasons. Overall, providing NetBSD-current helps us to create a more stable, accessible system.

It makes it easier for people to become involved in the development of NetBSD. Distributing the current development sources allows a greater number of people to see where the system is going, and to become involved with new features as they are implemented.

It also makes changes from users easier to integrate. If users make changes against the current development sources, then virtually no integration is needed to get them into the master source tree.

It also allows wider testing of the software as it is developed. Users of NetBSD-current are encouraged to send in bug reports about the current sources, and that helps find and fix bugs. Because people are testing the software soon after it's written, more bugs can be found and eliminated.

Installing a current snapshot (top)

To quickly begin using current, start with a snapshot generated by release engineering. The current status of each platform can be seen at NetBSD Autobuild and the corresponding releases found in ftp://ftp.NetBSD.org/pub/NetBSD-daily/HEAD/ by date and platform.

Hunt down to the desired binary/sets directory, and mget *.tgz files into your favorite local administrative directory (for example, $HOME/current); when limited by disk space and/or time, only kern-GENERIC, etc, base, and comp (if you want a compiler) are essential.

Extract the desired /etc and kernel:

    $ su
    $ cd /root
    $ tar -zxpf ~/etc.tgz
    $ tar -zxpf ~/kern-GENERIC.tgz
    $ ln -fh /netbsd /netbsd.old
    $ mv netbsd /netbsd
    $ shutdown -r now

Extract the matching base, and any other desirable feature sets:

    $ su
    # cd /
    # tar -zxpf ~/comp.tgz
    # ...
    # tar -zxpf ~/base.tgz

Update /etc as last step: postinstall whill first check and fix most things that can be automated, and etcupdate in the second step will ask on what to merge:
```
    # /usr/sbin/postinstall -s /root check
    # /usr/sbin/postinstall -s /root fix
    # /usr/sbin/etcupdate -s /root
    # shutdown -r now
```

At this point, you are relatively current and ready to build your own current source.

Downloading current source (top)

Traditionally, the system source files are kept at /usr/src, but this generally requires root privileges. The current build.sh process can run entirely unprivileged, although installation still requires root privileges. Whenever examples in this document assume /usr/src, you can substitute another location (such as $HOME/current).

Select a location for the source tree:
```
    $ cd /usr
    $ su
```
Download the -current source from a NetBSD mirror site near to you:
- via ftp from /pub/NetBSD/NetBSD-current/tar_files/src/, or
- using sup.
These files represent a snapshot of the source tree. For the most up-to-date files using anoncvs,
```
    $ cd /usr/src
    $ cvs -q -d $CVSROOT update -dP
```
The -d $CVSROOT is only needed on the first update, to populate the CVS tags with your selected mirror. Remember to always use the -P flag or add it to your .cvsrc file.

If you wish to track local changes to the NetBSD source you might want to setup a local CVS tree, and then import the sup changes.
Fix permissions
If you wish for the source tree to be maintained by a non-root user that is a member of the (traditional) wsrc group, do (as root):
```
$ chown -R user:wsrc /usr/src
$ chmod -R u=rwX,g=rwX,o=rX /usr/src
```

Building a release from source (top)

Please remember to check src/BUILDING for the latest changes.

Traditionally, the system object files were kept at /usr/obj, but this generally requires root privileges. Alternatively, keeping the object files on another filesystem can significantly speed compilation time. Whenever examples in this document assume /usr/src, you can substitute another location (such as $HOME/current).

Select a location for the object tree, where there is enough space for a full install, plus a set of release tarfiles:
```
    $ cd /usr/src
    $ su
    # mkdir ../tools
    # mkdir ../obj
```

From the root of the source tree:

    $ cd /usr/src
    $ ./build.sh -O ../obj -T ../tools -u -U release

In this example, the -u option indicates that a make clean operation should not be run before starting the build. This is useful when doing an update from a previous build and/or a fresh build.

The -U option allows the entire build by a non-root user.

When completed, you should have everything you need to install in a directory that build.sh selects (and will display), including install media and notes.

If you wish to cross compile for a different architecture, include '-m MACHINE -a ARCH' when running build.sh.

For more details, run './build.sh -h', and see /usr/src/BUILDING.

Updating an existing system (top)

Please remember to check src/UPDATING for the latest changes.

From the root of the source tree:
```
$ cd /usr/src
```

Build the toolchain:

$ ./build.sh -O ../obj -T ../tools -U -u tools

Build the distribution:

$ ./build.sh  -O ../obj -T ../tools -U -u distribution

Build the kernel:

$ ./build.sh  -O ../obj -T ../tools -U -u kernel=GENERIC

Install the kernel:

    $ cd ../obj/sys/arch/<ARCH>/compile/GENERIC
    $ su
    # mv /netbsd /netbsd.old
    # cp netbsd /netbsd

Reboot into the new kernel:
```
# shutdown -r now
```

Install the new userland:

    $ cd /usr/src
    $ su
    # ./build.sh -O ../obj -T ../tools -U install=/

Follow the instruction in the output for fixing obsolete files, for example:

    # /usr/src/usr.sbin/postinstall/postinstall -s /usr/src -d // fix defaults mtree obsolete

Update /etc:
```
# /usr/sbin/etcupdate -s /usr/src
```
Optionally reboot to ensure all running services are using the new binaries:
```
# shutdown -r now
```

In this example, the -u option indicates an update process, and the -U option allows the entire build by a non-root user followed with an install by root.

The build order (tools, distribution, kernel) is chosen to optimize the time for updating the source whenever problems occur. To ensure consistency, the process should be restarted from the beginning in the case of errors/cvs updates.

For more details, see /usr/src/UPDATING.

Things you need to remember (top)

When upgrading to a more recent version of -current you should always compile and boot a new kernel before installing any new libs (*). In general the best approach is to try the new kernel before anything else, and if you hit any problems see the entry in the Kernel FAQ.

Once the kernel is running, you should have a look at the BUILDING. file at the base of the source tree, and use the build.sh script to build a new userland.
When compiling a -current kernel, always remember to include the COMPAT_<lastrelease> option (e.g., COMPAT_16). As current diverges from the last stable release, compatibility code will be added, but it will only be enabled if this option is present. At a bare minimum, you will need this compatibility code for the time between booting the new kernel and finishing your build via build.sh
People using NetBSD-current are strongly encouraged to subscribe to the current-users mailing list. The source-changes mailing list is also of interest.

*: Unless you are certain there have been no new system calls added, but do it anyway; it's safer.

What are the various Makefile targets? (top)

For further documentation concerning usage of the new toolchain through the script 'build.sh' (in the toplevel source directory), run './build.sh -h', and see /usr/src/BUILDING.

WARNING: The usage of 'make build' has been deprecated by the updated toolchain, and is strongly discouraged.

When you build your system for the first time using build.sh, a set of tools for future use of compilations will be built, too. Any subsequent compilation should reuse the already compiled tools, and thus take less time.

Of course, don't invoke ./build.sh install=/ unless the ./build.sh build has succeeded previously or it's entirely possible to end up with a non-working system.

Using anoncvs (top)

These instructions cover unencrypted anoncvs connections. If you wish to use encryption protocols, see below.

Install devel/cvs. If NetBSD is built from -current sources past 2000-09-04, cvs is already installed.
Set the CVSROOT environment variable to point to the anoncvs server of your choice:
- For csh(1) or shells/tcsh users:
```
# setenv CVSROOT :pserver:anoncvs@anoncvs.NetBSD.org:/cvsroot
```
- For sh(1), ksh(1), or shells/bash2 users:
```
$ CVSROOT=:pserver:anoncvs@anoncvs.NetBSD.org:/cvsroot; export CVSROOT
```
```
$ cd /usr
$ cvs login
```
(use password "anoncvs")

You have to have write permission on the directory for the initial checkout; after that you can just change the owner of the source tree to some other user. One of the possible ways is to do the initial checkout as root, and then give the source tree to a different user for later use.

Using anoncvs over ssh (top)

The methods described in using anoncvs can be used over ssh to ensure the integrity of the sources you receive. However, this adds substantial overhead to the anoncvs servers.

Those servers in the mirrors that support ssh connections show the required information with each entry.

In general, remove the ':pserver:' prefix on the cvsroot, and set the variable CVS_RSH to 'ssh', using the method appropriate for your shell.

Tracking NetBSD-current with anoncvs (top)

Setting up

To checkout only the kernel sources
```
$ cd /usr
$ cvs checkout -P src/sys
```
This gives you the kernel sources in /usr/src/sys. Information on how to build a kernel is also available.
Checkout the entire source tree (including kernel)
```
$ cd /usr
$ cvs checkout -P src
```
You should now have a full set of NetBSD sources in /usr/src.

Note

It is almost always faster for a first-time "whole source" checkout to FTP the tarballs and untar them locally because that makes best use of the network link. After that, using cvs checkout/update works to minimize the number of bytes coming over by sending only the changes.
Fix permissions
If you wish for the source tree to be owned by a non-root user, do (as root):
```
# chown -R user /usr/src
```

To update the sources

To update only the kernel sources
```
$ cd /usr/src/sys
$ cvs update -dP
```
To update the entire source tree
```
$ cd /usr/src
$ cvs update -dP
```

Note: Running cvs checkout -d dir src (or similar commands with the other src* modules) does not work. You will get error messages saying "existing repository ... does not match ...; ignoring module _gnusrc-cmp" etc. The workaround is to drop the -d option and let cvs create the default directory.

To check out the sources from a certain date

$ cvs checkout -D 20020501-UTC src

To check out the sources from a certain branch

$ cvs checkout -rnetbsd-1-6 src

See src/doc/BRANCHES for a description of the branches in the CVS repository.

Useful hints

Do not use the cvs '-z' flag. The data stream gets out of sync, leading to corruption on the client, or causing the client to hang completely. The additional load is also hard on the cvs server.
If you want to check out a certain branch of the tree, you may want to take caution not to overwrite any existing directories by creating a new directory for this branch:
```
$ cd /parent/dir/to/checkout/into
$ mkdir NewName-temp
$ cd NewName-temp
$ cvs checkout ... src
$ mv src ../NewName
$ cd ..
$ rmdir NewName-temp
```
You will have to use objdirs in order for cvs updates to work correctly. If you happen to get errors from cvs saying things like:
```
   cvs [update aborted]: could not chdir to gnu/usr.bin/gdb/gdb: Not a directory
```
you should do a make cleandir and try again. Make sure to run make obj after the cvs update.
You can put switches for specific commands in a .cvsrc in your home directory, and they will be automatically used. A sample .cvsrc would be:
```
 update -dP
   checkout -P
   diff -u
```

Building NetBSD from source

(assuming you have an up-to-date NetBSD binary snapshot, and source in /usr/src, on your machine already; further assuming your BSDOBJDIR should be /usr/obj):

To build userland the first time:

# mkdir /usr/obj
# cd /usr/src
# ./build.sh -O /usr/obj -D /usr/NetBSD-new-build -T /usr/tools build
# ./build.sh -O /usr/obj -D /usr/NetBSD-new-build -T /usr/tools install=/

When you build your system for the first time using build.sh, a set of tools for future use of compilations will be built, too. Any subsequent compilation should reuse the already compiled tools, and thus take less time.

Of course, don't invoke ./build.sh install=/ unless the ./build.sh build has succeeded previously or it's entirely possible to end up with a non-working system.

To update userland binaries after a CVS update:

# cd /usr/src
# ./build.sh -D /usr/NetBSD-new-build -O /usr/obj -T /usr/tools -u build
# ./build.sh -D /usr/NetBSD-new-build -O /usr/obj -T /usr/tools -u install=/

These will install the new binaries on the running system - reboot to make sure they all take effect.

If you update system frequently and want the build to directly update your running system, you can use expert mode and build with DESTDIR=/, eg:

# ./build.sh -E -O /usr/obj -T /usr/tools -u build

Note this is for expert users only and you can very easily render your system into state where it won't be able to compile anything anymore. Use only if you are sure the build will finish successfully.

Tracking NetBSD-current with SUP into CVS (top)

Overview

Current can be tracked in the following way. The baseline copy of the sources is kept up to date using sup approximately once a week. as normal. This baseline source tree is then imported into a local CVS repository. Current is then built from a checked out copy of the repository.

There are 3 major reasons for this approach

It keeps track of how current changes over time.
It allows for local changes to be almost automatically merged into the updated current sources.
It ensures there is always a clean unmodified copy of the NetBSD-current source tree is available in case of problems when building.

The only downside to this approach is that 3 independent copies of the source tree are needed which amounts to about 150MB of disk space not including the space required to actually build current.

Requirements

CVS 1.9 or later (either already installed if you're running -current after 2000-09-04, installed from pkgsrc or just built from source). CVS 1.10 or later is preferred as it handles merging better.
SUP installation
Perl 5 installation for supplied script (optional)

Details

Tracking and building current consists of 6 phases:

Supping updated sources into master source tree.
Importing supped sources into CVS and updating working copy of sources.
Merging supped sources with local working sources.
Building and installing current.
Tagging the sources for a successful build in the repository.

Supping sources (top)

Sources can be supped from any NetBSD sup server and the output from the SUP should be stored in a file for later reference.

Importing and merging sources. (top)

Sources are imported as follows:

$ cvs -d /misc/cvsrep import -I ! -I CVS netbsd netbsd current-date

date is replaced by the date of the SUP for tracking purposes. The -I ! -I CVS options ensure that no file in the source tree is ignored except 'CVS' directories. This is because some NetBSD source files have extensions which are normally ignored by CVS. If there are any conflicts with local patches the import command will report them and will describe a command to merge the conflicts something like:

$ cvs checkout -jnetbsd:yesterday -jnetbsd netbsd

This merge command will correctly merge the imported NetBSD sources but it will not handle the removal of files locally which have already been removed by the SUP process. To do this the merge command would be:

$ cvs update -jprevious import tag -j current-date

previous import tag should be replaced with the name of the tag used for the previous cvs import. date should be replaced with the current date to yield the same tag used on the current import that has just been merged.

The conflicts reported by the import command are potential conflicts. These are usually merged by the update command but in some cases a real conflict occurs. In these cases a manual merge of the conflicting lines will be required. A real conflict will be reported in the cvs update output as a C followed by a filename.

Merging conflicts manually is not a simple process but in most cases it should be resolved by removing the local changes and making the file like the original NetBSD source code.

CVS marks conflicts as follows:

<<<<<<
  code from local file
======
  code from imported file
>>>>>> local revision number of newly imported revision

If the import reports no conflicts the checked out copy of the tree should be updated in exactly the same way as for the conflicts case.

All update and checkout commands should be done in the directory where the sources have been checked out. On my system this is /usr/src/netbsd.

If this is the first import then there will be no sources checked out. Assuming you wish to create the source tree in '/usr/src/netbsd' The following commands will check out the source and no merge step is required.

$ cd /usr/src
$ cvs -d /misc/cvsrep checkout netbsd

Building current. (top)

Configure, build, and reboot into a new kernel.
cd to the base of your -current source tree and type ./build.sh -T /usr/tools -O /usr/obj .
You may need to merge in any changes that have been made to files in /etc.

Tagging a successful build (top)

If the build completes successfully, and produces a working set of binaries, it can be useful to tag the working sources. This allows rewinding to a working build tree with a single CVS command in the event that the current tree becomes unbuildable for any reason. This can be performed by issuing the following command:

$ cvs tag successful-build-build date

Notes

If the NetBSD customised version of CVS, which recognises $NetBSD$ markers in files, is not used, the NetBSD revision number of the file is available for reference purposes when build problems occur.

The sup/import/merge sequence described above is quite easily automatable. The following Perl script automates this process.

#!/usr/pkg/bin/perl
#
# Script to SUP NetBSD-current, import it into CVS and merge it with
# any local changes.
#
# NOTES:
# This script does no error handling so is not really suitable for 
# non-interactive use.
#
# This script has only been test with cvs-1.10.1 and cvs-1.9.18.
#
$SRCROOT="/usr/src/netbsd";
$IMPORTROOT="/misc/import";
$CVSROOT="/misc/cvsrep";
#run the sup into a perl stream
system "/usr/sbin/sup -zsv" ; # This may need to change for none
                              # current systems

# now import the new files into CVS 

chdir $IMPORTROOT or die "Could not cd to $IMPORTROOT\n";

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
$date = localtime;
$shortdate = sprintf "%02d%02d%04d",$mday,$mon+1,1900+$year;
system "/usr/local/bin/cvs -d$CVSROOT import -I ! -m\"SUP Import $date\" netbsd netbsd current-$shortdate ";

# make the working directory the local NetBSD Tree
chdir $SRCROOT or die "Could not change to $SRCROOT directory\n";

# Now do the import.
$lastimport = `cat /usr/src/netbsd/.tag`; # `s are backquotes
$lastimport =~ s/\n//; # strip off any trailing newline in the string
system "/usr/local/bin/cvs update -j $lastimport  -j
current-$shortdate ";
# Now write the current file into tag save file
open TAG,">$SRCROOT/.tag" or die "Could not open new tag file";
 print TAG "current-$shortdate";
close TAG;

This script was written in Perl since it is the scripting tool which the author has the most experience with. It should be fairly straightforward to write a shell script to perform the same task.

Techniques for tracking current with CVS have been discuss several times on the NetBSD current-users mailing list. For alternative techniques try searching the NetBSD mailing lists.

If you have any comments or suggestions please send them to Mike Pumford <mpumford@black-star.demon.co.uk> (who maintains this entry) or <www@NetBSD.org>.

Getting the whole repository (top)

All the procedures described above allow you keeping your own changes in your repository, which has its advantages if you develop your own software based on NetBSD. If you don't want to maintain your own CVS repository, but just want to mirror NetBSD's CVS repository, there are three ways to do so.

Each of the methods described briefly below will get you a copy of the NetBSD CVS repository (i.e. the RCS ,v files, not the checked out files!). You can then setup your own anoncvs server or check out to a local harddisk. It's also useful for fast access to the history information stored in the repository.

The methods to retrieve the whole repository are:

sup

If you use sup already to mirror other parts of the NetBSD source, you will want to add the following lines to your sup config file:

anoncvs release=all  host=sup.NetBSD.org hostbase=/ftp/pub \
base=/usr prefix=/usr backup use-rel-suffix compress

After that, run "sup /path/to/supfile anoncvs" to retrieve the files.

Some example sup files are available in /usr/share/examples/supfiles. Also, check our list of SUP mirrors to find the server closest to you!

rsync

Note that rsync puts quite a heavy load on our rsync server, and as such the number of concurrent rsync users is restricted. If you still want to try rsync, the command to retrieve the repository is:

rsync -v -a rsync://anoncvs.NetBSD.org/cvsroot/src .

Please see our list of rsync mirrors!

cvsup

CVSup is not currently available for all NetBSD architectures, since the M3 compiler has not been ported. On i386, you can mirror the repository from cvsup.de.NetBSD.org with the devel/cvsup package and the following config file:

*default host=cvsup.de.NetBSD.org
*default base=/usr
*default prefix=/local/NetBSD-cvs
*default release=cvs
*default delete use-rel-suffix
*default compress

netbsd

Please see our list of CVSup mirrors!

What if I get an error? (top)

If you try to build -current, either from a snapshot or an earlier -current, and it doesn't work, don't panic. Try these steps:

Read the UPDATING file from the release you're trying to build.
Read the current-users archive for hints.
Update again. You may have caught the repository in the middle of a commit to several related files, or the problem might have already been fixed.
If all else fails, send email to current-users explaining the problem. Include the date, time, and method you used to get your -current sources, as well as any local changes you've made. Then put in a short script that includes the error messages you're getting. Somebody will probably fix the problem momentarily.

Updating the configuration and startup files with etcupdate (top)

Overview

etcupdate is a script to help users compare, merge and install new configuration and startup files (files found in the etc.tgz distribution set) in /dev, /etc and /root after performing an operating system upgrade. The upgrade of the operating system could have been performed either by compiling sources or by extracting the distribution binaries.

Using etcupdate with source files

In case where the sources are in /usr/src the following command should be enough:

# etcupdate

But what if your NetBSD sources are in an alternative location, such as in /home/jdoe/netbsd/src? Don't worry, tell etcupdate the location of your source tree with -s srcdir and it will work just fine:

# etcupdate -s /home/jdoe/netbsd/src

Using etcupdate with binary distribution sets

Sometimes it's not possible have the sources around but you still want to update the configuration and startup files. The solution is to extract the desired distribution files (at least etc.tgz) and use the -b srcdir switch to tell etcupdate that we don't have the sources but only the official distribution sets.

# mkdir /tmp/temproot
# cd /tmp/temproot
# tar xpzf /some/where/etc.tgz
# etcupdate -s /tmp/temproot

Specific problems

Console dead after updating to wscons (top)

You should copy a current MAKEDEV from the appropriate etc.port directory in src/etc, into /dev, boot single user, then type:

# fsck -p
# mount -vt nonfs
# cd /dev
# ./MAKEDEV wscons

Why does build.sh always rebuild nbmake first? (top)

Even after running ./build.sh tools and using the -u flag or specifying TOOLDIR in /etc/mk.conf, nbmake is always rebuilt by build.sh. This is normal. The reason for this can be found in ./build.sh itself, in the function rebuildmake:

        # Note that we do NOT try to grovel "mk.conf" here to find out if
        # TOOLDIR is set there, because it can contain make variable
        # expansions and other stuff only parsable *after* we have a working
        # ${toolprefix}make.  So this logic can only work if the user has
        # pre-set TOOLDIR in the environment or used the -T option to
        # build.sh.
        #

So, if you do not want to rebuild nbmake, you will need to pass -T tooldir or set the TOOLDIR variable in the environment.